ngi-search
27 de mayo de 2024
The NGI Search project, co-funded by the European Union's Horizon 2020 research and innovation program, has unveiled the outcomes of its fourth Open Call.
This initiative aims to assist diverse beneficiaries, including researchers, developers, internet activists, hackers, SMEs, and startups, in adopting and developing solutions for data and resource discovery on the Internet.
109 proposals were submitted for this call, resulting in the selection of 13 proposals that have sought financial support. These beneficiaries will receive technical and business support, from Open Source Licensing to Market Readiness.
👉 More information here.
✨Meet the selected projects.✨
✅ Datami
The Datami project is aimed at small or mid-sized organisations within civic society (NGOs, associations, research groups) that provide public information. Such entities often produce many valuable datasets but need more technical expertise, time, and resources to transform them into discoverable and reusable data.
🔥Despite their crucial role in elevating public debate, they remain under-recognised.🔥
Datami (100% open) is designed to simplify the preparation and valorisation of datasets for immediate and further use, especially for teams with limited resources. With Datami, raw datasets are structured into standard formats (e.g., Data package), versioned with Git, data-visualised, and published on dedicated open data catalogues (e.g., CKAN). It also ensures data producers control data storage and permissions (read/write/contribute).
Structure, enriching, and making raw datasets interoperable will allow more civic actors to produce digital commons.🚀
✅ Fediverse Discovery Provider
Today, each application (and server/instance) participating in the Fediverse is independent, mainly regarding discovery. User search is possible via the WebFinger protocol, but content, trends and recommendations are not easily shared across the network.
✨This initiative builds a reference discovery provider and protocol (and an example consumer implementation inside of Mastodon) to enable users to discover content across the rich diversity of the fediverse.✨
🔏 The implementations and protocol will respect user privacy.
Discovery providers should be "pluggable"—servers should be able to choose from one or more, in line with the network's decentralised and federated nature.
✅ COGNITREK, Driving Cognitive Accessibility
COGNITREK is an innovative initiative focused on creating a system for individuals to understand complex documents via a conversational interface. Going beyond classic web accessibility, the project aims to enhance cognitive accessibility, breaking down barriers to understanding information for a more inclusive digital experience.
The Ximdex platform's AI-driven systems have already automated digital content to provide web accessibility through, among others, relevant entity recognition, image captioning, and text translation.
Yet, a persistent challenge remains—adapting a text to individual readers, particularly in educational settings.
✨COGNITREK's primary objective is Cognitive Accessibility, achievable through Decoupling Information, Providing Context, and AI-based Conversation and Dialogue.✨
Their mission is to create a seamlessly inclusive digital landscape, revolutionising how people access and comprehend information.🚀
✅ Deep Open Data Search
Many public institutions publish datasets using open data platforms (such as CKAN or Socrata).
However, these platforms could be more robust for most users. Datasets are often poorly described, so it's hard for a user to know which one has the relevant data they need.
When using standard search engines, more than these descriptions are needed to pinpoint a specific dataset, and the user is forced to sift through many irrelevant results. Organisations use technical jargon or internal terms, making it hard to use. In most cases, data itself is provided without schema or field documentation.
This initiative proposes a novel solution, using LLM and RAG capabilities (along with more "classic" data processing methods) to:
📌 Allow deep querying and analysis of singular data points, enabling uses such as answering research questions or fact-checking.
📌 Improve the quality of open data publishing platforms.
📌 Enhance dataset findability for classic search engines.
✅ Carbon.txt
Carbon.txt is an open-source project that helps internet users discover structured and comparable sustainability data, specifically focusing on energy data and its greenness. With this proposal, the project will expand carbon.txt to include energy efficiency information mandated by the EU as part of the Green Deal.
Carbon.txt is an ideal vehicle for discovering this information. It is an existing open convention that enables anyone on the internet to freely disclose how their digital infrastructure runs on green electricity in a machine-readable way. It is inspired by tools like security.txt, which supports search engines in indexing information and leverages internet protocols and DNS.
💫To their knowledge, this is the only project looking to connect data in such a way.💫
This project unlocks the potential for every internet user to discover and compare critical information about large companies where that data has previously been inaccessible, non-standardised or locked behind paywalls.
✅ AI-Generated Code Search
Generative AI engines and large language models (LLMs) are emerging as viable tools for software developers to automate code writing. These engines and LLMs are trained on publicly available, free, and open-source (FOSS) code.
AI-generated code can inherit the license and vulnerabilities of the FOSS code used for its training. Therefore, identifying AI-generated source code is essential and urgent, as it threatens the foundation of open-source development and software development and raises major ethical, legal, and security questions.
Based on their experience creating industry-leading FOSS code origin analysis tools for license and security, they designed a new approach to identify and detect whether AI-generated code is derived from existing FOSS with a new code fragment approximate similarity search.
This proposal will deliver a reusable open-source library and the indexing code to create an open dataset that identifies whether the source code is AI-generated and reports which FOSS project it derives from.
✅ Eclipse Data Components (EDC) Search Engine
Eclipse Data Components (EDC) Connector is open-source software used by 1,800 companies, such as BMW, France TV, or Air France, to exchange data with other companies. This project extends EDC Connector with decentralised data search and discovery to simplify sharing.
Today, companies can only share data one-on-one. To do so, they must copy-paste data using the Eclipse Data Components (EDC) Connector. Sharing B2B data is, therefore, costly—around 800€ per data set.
🔥Companies lose control over the data once it's shared, and it's impossible to discover it.🔥
This project extends the EDC Connector to enable indexing, searching, and discovering decentralised data among B2B partners. This will reduce the cost of sharing data to a couple of minutes per dataset, enhance sovereignty over the shared data, and enable B2B data discovery.
They've been developing INRIA, a state-of-the-art decentralised data search engine, for two years and are active members of the Data Space ecosystem, so they deeply understand the need for a better way to discover and search data from B2B partners.
✅ Confidential Private Search (COPS)
Organisations commonly outsource their security monitoring to third-party managed security services or threat intelligence search tools. Today, such services and tools do not offer confidentiality protection to data, potentially exposing sensitive personal information.
COPS enables private indexing and search of threat intelligence information in machine-based data. It combines the rich functionality of OpenSearch with the confidentiality, integrity and trust assessment primitives of confidential computing.
✅ Trallie
Trallie ("Transfer learning for information extraction") boosts IE for search among textual asset descriptions by eliminating costly human annotation and leveraging LLM capabilities to follow NL guidelines, understand labels, and manipulate NL as it does for code.
NL descriptions of assets and resources are here to stay as legacy or as flexible catch-alls. Clustering and categorising them to run structured search queries traditionally requires IE (RAG and dense embedding matching offer some partial solutions).
🔥 This often needs to be improved by costly human annotation, if only by providing few-shot examples of categories.🔥
Trallie brings the transfer learning and world understanding afforded by LLM to make IE agile.
They deliver multilingual, IE-fine-tuned checkpoints of various open model architectures and, for reproducibility, our full FT recipe, including prompt templates.
Transfer learning and NL input imply impacts on legacy and low-resource scenarios, improving the discoverability of hidden asset collections, the plurality of sources through more accessible access to search tools, and improved trust and privacy.
At Pi School, their experience of rapid prototyping in AI, acquired over more than 100 AI projects, giving them an advantage in exploiting the rapidly moving SOTA.
✅ On My Disk: search integration
On the current Web, content generation is decoupled from search:
📌 While end users are primarily responsible for creating documents, search systems are owned by separate entities that control how content is indexed and found.
📌 For individuals, this results in weak control over the indexing of their shared content and privacy issues at search time.
✨The On My Disk (OMD) and PeARS projects offer a private cloud solution with a tightly integrated search engine.✨
When a user modifies their content, the decentralised search engine updates its index, respecting any restrictions the user imposes. No third-party search engine is involved.
They now extend this setting with a one-click Web hosting module. A user's website simply sits in their OMD shared folders but is rendered under a permanent URL on the Web. The result is a blueprint for a novel Web architecture where indexing and search fall out of content creation, returning complete control and privacy to end users.
✅ Ensuring Fairness in Democratic AI
Eticas.ai is a Spanish company that produces performance measurements of protective attributes in data-driven decision-making systems.
For the last 12 months, the project team has been working on producing AI audits as part of the AI auditing team at Eticas.ai, a spin-out of the Eticas Foundation. In this time, they have made remarkable progress and now have a software platform that allows us to measure the performance of protective attributes in any data-driven decision-making system and compare model data with their demographic libraries.
They focus on identifying how specific, protected, and non-protected outliers perform throughout the data and decision-making lifecycle to ensure that algorithmic outcomes and impacts do not further discriminate or marginalise individuals and groups.
✅ MOUSSE-Metadata fOcUsed Semantic Search Engine.
MOUSSE is developing a semantic search engine that uses large language model (LLM) technologies. This engine allows users to search using free-text queries to find data assets based on their metadata. The project aims to enhance the user experience and effectiveness in accessing large and varied databases that do not have a clear ontological structure.
✨The developed technology is versatile and can be applied across various themes and sectors within established data portals and repositories.✨
This new system builds on the knowledge and experience gained from the EIFFEL H2020 project, which also focused on LLM-based semantic searches, specifically for Earth Observation (EO) data. MOUSSE plans to introduce new features and improve the technology readiness level (TRL) from 5 to 7 to prepare for practical application.
The GEOSS Portal's data will serve as a significant testbed for this technology, especially for emerging EO applications field researchers.
Finally, the project aims to enhance the global accessibility, sharing, and use of EO data, supporting initiatives like the Green Deal and the Paris Agreement🚀
✅ Neural Datafari
This initiative proposes an ambitious end-to-end open-source search solution that leverages vector search, RAG, multi-sources, and multi-formats.
This will allow citizens and organisations to easily leverage work done on LLMs and the growing diversity of sources. Only a RAG approach allows leveraging pre-trained LLM models in citizens and enterprise scenarios.
This project also proposes improving Apache Solr 9's early neural search capabilities and integrating them in Datafari to enable citizens to benefit from neural search with an open-source end-to-end solution, from document fetching to the search via a REACT-based user interface. The retrieved documents will be injected into an LLM for a proper RAG for the user query.
The project will propose a docker version of the solution to facilitate citizens' and organisations' use. 100% of the developed source will be contributed to Apache Solr, Apache ManifoldCF, and the Datafari Community.
🫵 Ready to be the next NGI. Search trailblazer?
✨The NGI Search 5th Open Call is opening on the 29th of May, so you can join us in this journey to redefine the future of search and discovery!✨
The NGI Search project is an endeavour for all who can convincingly demonstrate their commitment to transforming how we utilise, experience, explore, and uncover data and resources on the Internet and the web.
Remember to join the NGI Search Community and keep you updated by the project's Twitter (X) and LinkedIn for the latest news.
