Paris-based and female-founded AI startup Pathway has announced the general launch of its data processing engine. Reportedly, it is up to 90x faster than existing streaming solutions, and promises to be the “fastest data processing engine on the market.”
The secret? A unique ability to mix batch and streaming logic in the same workflow, which lets the system forget things that are no longer useful. Basically, this means it can learn and react to changes in real-time — like humans.
Traditionally, the complexity of building batch and streaming architectures has resulted in a division between the two approaches, says Pathway CEO and co-founder Zuzanna Stamirowska.
This, she adds, has slowed down the adoption of data streaming for AI systems and fixed their intelligence at a moment in time. Not to mention the added complexity of a third workflow — generative AI.
According to Stamirowska, there’s now a “critical need” for rapid data processing and more adaptable AI. “That’s why our mission has been to enable real-time data processing, while giving developers a simple experience regardless of whether they work with batch, streaming, or LLM systems,” she states.
Revisions to data points without AI retraining
Machines “forgetting” incorrect or outdated information in real-time has been a near-impossible feat in the past, due to models being trained on static data uploads. Traditionally, unlearning would require retraining of the model.
Indeed, when we put the question to ChatGPT for fun — can you unlearn things should they prove to be inaccurate — this is the response we received:
“As an AI language model, I don’t have the ability to “unlearn” information in the same way humans do.
“However, the developers and researchers at OpenAI can update and retrain the model based on new data and improvements.”
But Pathway says it can make revisions to certain data points without requiring a full batch data upload, akin to updating the value of one cell within an Excel document. The updated cells doesn’t reprocess the whole document, but just the cells dependent on it.
One of the startup’s existing clients, German logistics specialist DB Schenker, reduced the time-to-market of anomaly-detection analytics projects from three months to one hour. Meanwhile, French postal services company La Poste saw a fleet CAPEX reduction of 16%.
‘Lingua franca’ for developers
Polish-French duo Stamirowska and Claire Nouet, the company’s COO, founded Pathway in 2020. Thus far, the startup has raised $4.5mn (approx. €4mn) in a pre-seed round December last year, and counts 20+ employees across Europe and North America.
The female-led deep tech startup is hoping for its system to become a “lingua franca” of all data pipelines (stream, batch, and generative AI). Beyond cutting costs for clients, it says it is looking to democratise the ability for developers to design streaming workflows, which have typically required a specialist skill set.
