proSapient offers a feature packed and modern platform to support primary search — from expert consultations and transcripts to powerful surveys.
We joined up with proSapient to help them build Darwin — a new expert self-sorting semantic search engine designed specifically to connect industry professionals to relevant projects using cutting-edge AI search and Machine Learning.
More frequently than ever, organizations are looking for independent industry experts and freelance management consultants to help them solve critical challenges they face on their road to business success. But, in the sea of experts across numerous industries and domains, finding the ones that perfectly match your business needs can be a real challenge.
proSapient approached us to help them address the opportunity and build a new expert search engine that would provide users with the ability to find better high-quality, differentiated experts, save user time through automation and enjoy stress-free project scalability.
I am extremely glad and honored that HTEC got a chance to work on this remarkably interesting challenge that can make an impact not just on our client’s business, but also on businesses of their clients’ as well. It is always a pleasure to work with a client that is so passionate about innovation, more so when that innovation can position you as a number one expert platform in the world. Combining passion for machine learning and engineering excellence yielded the next iteration of an expert recommendation engine, called Darwin, that will improve both daily routines of proSapient’s consultants and the experience of their clients. We worked as a team, and that is always a recipe for success.
– Nikola D. Stanković
Senior Engineering & Delivery Manager at HTEC Group
To help users find the best worldwide industry experts, we collaborated on a solution which ingested large volumes of data to make expert recommendations.
We standardized and enriched the company data and job data models with additional company information coming from all experts’ profiles, including their history, company data, job title as well as different parts of the platform by integrating with third-party data providers, as well as our own. This allowed us to build one rich company data model. A similar process was conducted with job titles. Standardized data was indexed ahead of a match.
On top of this, we also introduced semantic search. Here we leveraged the power of Natural Language Processing techniques to be able to use the text that comes as a part of the experts’ biography. The Machine Learning part of the system focused on these two NLP tasks:
Creating semantic representations of textual data powered by Universal Sentence Encoder (USE) which turns text into vectors.
When a text becomes numerical, it allows us to gain different perspectives while preserving the semantic meaning of the content.
This allowed us to create the vector representation of the experts’ biography which keeps the semantic meaning of all the words. The same process was conducted with the description of the projects and all their aspects. On top of this, we also conducted the standardization of geolocation.
When there is a certain requirement coming from a client, the matching is conducted based on this standardized and clean data which is now much better structured and not based on keywords only, but semantics as well. This means that not only the right word is matched but the context as well.
Understanding structural elements in textual data sources powered by Core NLP, Stanford model.
This model recognizes entities in words, meaning it can recognize organizations, the name of a company or geography in some text. Then it simply uses this data, especially data related to job positions and senioritis allowing us to make automatic categorization, and create models used for the prediction of roles and senioritis for job titles that originally do not have this data.
We also used a semantic approach to perform the clustering of job titles, and then standardize them with unique titles and names. This further helped us decrease the number of unique job titles as there were too many of them.
It never stops to amaze me how much tedious work there actually is in daily routines of people working in data-centric environments. proSapient consultants weren’t an exception to this non-written rule. Our motivation behind all the effort was to make their job more efficient and focused on what really matters – making their clients’ experience a positive one. By standardizing data, integrating with powerful data providers, building company and job titles rich data models, and finally empowering the platform and matching logic with intelligent NLP processing solutions, we’ve hidden all the mind-numbing work away from the consultants and made their job easier. Darwin is doing the core work now, enabling the consultants to put their attention on experts and clients, rather than data. By shifting their focus on core business cases, we’re helping proSapient build a powerful workforce and place themselves on a global map as a leading research platform for the private equity and consulting sectors
– Sanja Bogdanović Dinić
Engineering and Delivery Manager at HTEC Group
From the beginning of the project, it was clear that HTEC wants to understand the context and the purpose of the product, the end-to-end processes and our goals, not just for the search purposes, but to be able to have a clear view of the entire business. I think that HTEC’s holistic approach to our project is one of the key points I would highlight. Second, the personal relationship we developed between two of our teams is truly unique. I always felt as if we all belonged to the same team, sharing the same vision and goals. And the third thing I would single out is the HTEC’s ability to handle pressure and tight deadlines and support us every step of the way no matter how complex the project requirements are.
– Luca Simon
Product Owner at proSapient
HTEC Group team has brought the capabilities of proSapient search engine to a whole new level by empowering it with AI technologies that drive better and more relevant search of the highest-quality experts. At its core, Darwin trains data, making it a key enabler behind AI’s understanding of both content and user intent.
This AI-fueled approach will help the search engine improve continuously and better understand the user’s intent and what is relevant to them providing data specific to each user.
Natural language processing (NLP) and machine learning (ML) also played a major role here as they helped automate the extraction, tagging and classification of concepts and entities within massive amounts of enterprise content.
What’s so revolutionary about Darwin is that it:
With Darwin, now it’s very easy for our internal team to validate every single search result. The quality of the search result improved enormously, but from UX perspective, it also became a lot easier and smoother to validate these results. So even the human element is a lot more efficient and advanced. The fact that we are now able to combine our internal data with external data sources technically reduces the need to do research outside of the platform. All of the data is served in one place— it’s revolutionary. I think this feature takes this product to the next level because it is creating something intelligent out of it — it’s a game changer.
– Luca Simon
Product Owner at proSapient
The partnership with HTEC was exceptionally good mostly due to the outstanding project management that was led by Sanja and Nevena. We built trust and could always rely on the team that they will give their maximum to deliver best results. And the support that we got, even after closing the official development cycle of the project, was very good. We benefited from this sort of approach and the close collaboration continued even after the product was handed over to the in-house team. I think the collaboration and the project itself were unique because we worked in a blended team — all the project requirements and the whole planning was shared between HTEC and ProSapient. I believe this highly collaborative approach was the key to building such an impactful solution like Darwin.
– Luca Simon
Product Owner at proSapient
