Apple Search is Already Here

by Posted @ Sep 10 2020

Twitter

Apple Search Starts with Siri Search

In 2012, I wrote about the Apple Siri Patent application. The patent I wrote about was Intelligent Automated Assistant. In coming out with Siri, an automated assistant, Apple was creating something much more complex than just a search engine. As the patent filing says:

Unlike search engines which only return links and content, some embodiments of automated assistants described herein may automate research and problem-solving activities.

The Siri patent tells us about the Active Ontology approach that Apple adopted for use with Siri.

It describes how “Active Ontologies” serve as the infrastructure behind the intelligent assistant building models about data and knowledge. An example from the patent tells us that a “dining out domain model” might be linked to a “restaurant concept” as well as a “meal event concept.” The way information that the intelligent assistant uses is the model behind how Siri Works.

In 2015, I wrote Here Comes Applebot: The Start of Apple Web Search? Apple had purchased Topsy, which had been developing an expertise in Search. Topsy appears to have been a strong part of search at Apple, with people from Topsy becoming employed at Apple Search. And search patents from Topsy becoming intellectual property at Apple

Apple Search Engine Job Post

A present-day job application at LinkedIn for a position of ML Research Manager, Siri Search Relevance tells us about the state of search at Apple, or where they would like search to be:

The Siri Search Relevance team is creating groundbreaking technology for artificial intelligence, machine learning, and natural language processing. The features we create are redefining how hundreds of millions of people use their computers and mobile devices to search and find what they are looking for. Siri’s universal search engine powers search features across a variety of Apple products, including Siri, Spotlight, Safari, Messages, and Lookup. As part of this group, you will be leading large scale machine learning and deep learning to improve Query Understanding and Ranking of Siri Search and developing fundamental building blocks needed for Artificial Intelligence. This involves developing sophisticated machine learning models, using word embeddings and deep learning to understand the quality of matches, online learning to react quickly to change, natural language processing to understand queries, taking advantage of petabytes of data and signals from millions of users, and combining information from multiple sources to provide the user with results that best satisfy their intent and information seeking needs.

That Apple Job posting provides more details on what a person might do in this role, which tells us more about how search related to Siri might work:

Description

As a member of our fast-paced group, you’ll have the unique and rewarding opportunity to shape upcoming products from Apple. This role focuses on leading the development of algorithms and ranking systems that drive the quality experience of Siri Search features that are integrated throughout Apple products, including Siri, Spotlight, Safari, Lookup, Messages. You will lead and grow a team of data scientists, machine learning specialists, and infrastructure engineers. This role will have the following responsibilities:

* Work with product to define requirements for search ranking

* Analyze, measure, and evaluate the search ranking quality in the production system

* Define the architecture and roadmap of search ranking

* Identify and define search ranking projects, including ranking algorithms, the training and evaluation system, and the runtime ranking stack

* Lead and drive

* ML modeling for search ranking

* Measurement and evaluation

* Ranking improvement iterations

* Runtime stack and offline system improvements

* Lead and grow the search ranking team, which includes

* Data scientists that work on modeling, ML, NLP

* Engineers that build runtime ranking system and offline pipelines

* Work with partner teams to build high-quality Siri Search products

So search at Apple involves more than running a search engine

Apple Search Technology

The post I wrote about Applebot five years ago was about the search technology that Apple acquired from Topsy. I did a search a couple of weeks ago on search related patents at the US Patent and Trademark Office. One of the patents that I saw in my search was one that I thought was worth writing about because it covered web search.

It is specifically about deep links that might be found in an application, and how Apple might index those.

The description of the patent tells us it
is about:

  • Search technology
  • Indexing web pages with deep links
  • Improved proximity scoring function for query results
  • An improved federated query result ranking method

Someone performs a query search to look up information on the Web or from other data sources.

A search server receives a query string and searches a search index for results that match the query string. The indices being searched may include:

  • Web pages
  • Other objects available across a network
  • Information about objects located locally on the device (e.g., files, media, contacts, and/or other types of objects stored locally on the device).

Accessing one of these web pages or local objects will load the web page in a web browser.

Some web pages or other objects include deep links that are a reference to a location in an application, such as a location in a mobile application executing on a mobile device.

For example, the web site could be a review-type web site that includes thousands or millions of reviews for business, services, and/or other sites of interest.

One or more of the reviews hosted on the web site could include a deep link for that review.

A deep link allows a user to access content in a mobile application, where the mobile application would likely give a richer experience for this content than the user has when viewing the same content through a web browser.

It could be a richer experience because the mobile application has been developed to handle this content, where a web browser has been a more generally designed application.

Apple search seed objects

Deep Linking and Apple Search

Someone performs a query on a mobile device

It may determine that there are a number of results that match the query (a string match, in this instance.)

The device may determine that there are a number of results from native applications installed on the device.

For those device applications, the SERPs may include a link for each of the results that launches the corresponding native application if that link is selected, where the link is a deep link that references a location in the corresponding native application.

If one of those deep links is selected, the device launches the corresponding native application with the data for that link.

We are also told that a device may index deep links in a search index.

A search query may return deep links that have been indexed and are returned in response to a query.

This Deep Links Apple search patent can be found at:

Indexing web pages with deep links
Inventors: Jason Douglas and Vipul Ved Prakash
Assignee: Apple Inc.
US Patent: 10,755,032
Granted: August 25, 2020
Filed: September 30, 2015

Abstract

A method and apparatus of a device that performs a search using a deep link index is described. In an exemplary embodiment, the device receives a query on a device. The device additionally determines a plurality of results matching the query. The device further determines a subset of the plurality of results that correspond to at least one native application installed on the device. In addition, the device presents a link for each of the results in the subset of the plurality of results with data that launches the corresponding native application if that link is selected, where the link is a deep link that references a location in the corresponding native application. Upon detecting one of the links corresponding to the subset of the plurality of results is selected, the device launches the corresponding native application with the data for that link.

More About The Inventors

One of the inventors, Vipul Ved Prakash, tells us that this was his role at Apple Search:

Along with the incredibly talented Topsy team, I manage the search engines that power Spotlight, Safari & Siri on iOS and macOS.

He was also one of the Founders of Topsy:

Topsy was a social media search (consumer) and analytics (enterprise) engine, that was unparalleled in the depth of access and level of insight it provided. Among our achievements was an indexing system, written from the ground-up, that indexed over a trillion social posts, was completely real-time and provided instant search and analytics over Twitter. Customers included fortune 500s, movie studios, presidential campaigns, almost all top News companies. It was a great ride. I got to wear many hats from raising $27M in funding to recruiting an A+ team to push the envelope on technology and product.

Topsy was acquired by Apple in 2013.

Federated Search at Apple

The patent tells us that this search process is one that uses a federated search.

What that means is that a query search uses multiple search indices and a broad-based search index.

The multiple search indices may include specialized search indices, such as maps, online encyclopedia, media, sites, and/or other types of a specialized search index that are used to search specialized search domains

It may receive results from the multiple search indices and the second set of results from the broad-base search index.

We are told that each of the results are scored and the second set of results is ranked

More About Deeplinks

We are told what a deeplink is under this patent. The deep link is a Uniform Resource Identifier (URI), where the app_id is the application identifier, and the location is a location within the application, and info is further information that is used by the application to identify the location within the application. The application identifier may be known as a schema.

The patent tells us that many different kinds of application identifiers are possible, such as application identifiers for Twitter, YouTube, eBay, Netflix, Quora, Starbucks, Pinterest, Yelp, Etsy, and other applications.

Each of the applications has its own format for a deep link.

For example, a Twitter deep link is “twitter://location/ . . . ” and a YouTube deep link is “youtube://location/ . . . “.

The deep link crawler crawls the web or other objects to index discovered deep links.

The deep link crawler indexed deep links from on objects and stores the application identifier and the deep link (e.g., the URL for the deep link).

The patent tells us that a search engine may include a deep link search service.

That deep link search engine returns results that include deep links from the search index.

A Deeplink Indexer Indexes Content from Inside Applications

The patent tells us that by building the deep-link index, it is also building an index for content inside the application that corresponds to the deep link.

For example, if the object is a review of a business on a web page for a review web site and this web page includes a deep link to an application corresponding to the web site, the process indexes this deep link for the review of the business in this application.

This process may also determine if this object (the review web site) includes references to other objects than can be crawled. Other deeplinks may be URLs pointing to other objects that can be crawled, the process adds the references to these objects in the group of objects.

Proximity Scoring

Among other factors, search results are scored using a proximity-scoring function.

In one embodiment, the proximity scoring function scores the result where each result includes two or more terms from a query that are within a specified distance.

A term is a word, phrase, or another language construct. The distance between the terms can be characters or words and is used to constrain the matching of the terms for a result.

The proximity-scoring function imposes an order on the result.

For example, a query that is “white truck” would score the result higher for strings “white Ford truck,” “white slightly used truck,” but score lower “truck with white paint” (out of order) and “white house that includes a lawn with a truck” (longer distance between words).

The proximity of the terms in a document implies a relationship between the terms.

Creators of objects being searched may try to formulate sentences that contain a single idea, or cluster of related ideas within neighboring sentences or organized into paragraphs.

Thus, there may be an inherent or relatively high probability within this object structure that terms used together or in close proximity are related.

When two words are on the opposite ends of a document or web page, the probability of a relationship between the words is relatively weak.

By ranking results where the words are within the specified maximum proximity, or distance, the results are assumed to be of higher relevance than the results where the words are scattered.

Deep Links Apple Search Take-aways

Where things can be interesting in describing Apple Search is that we can make comparisons to search at Google.

Google has a support page where they describe how to set up deep links indexing at Firebase App Indexing

The Apple patent tells us about how they may index links in applications that they may come across such as Twitter. The Google support page provides instructions to app owners on how to get the content from their app indexed

Apple does also provide support information about getting content from your App indexed by Apple: Search Drives User Engagement

The last section of that page is headed, “Several Factors Determine the Ranking of Search Results.” These are factors that determine the rankings of the content found in Apps:

  • The frequency with which users view your content (which is captured only through the use of NSUserActivity)
  • The amount of engagement users have with your content, demonstrated when users tap a search result or find the information useful
  • In the case of marked-up web content, the popularity of a URL and the amount of structured data available

Apple doesn’t have a page that officially is referred to as a search engine, such as Google or Bing, but they have ways to search for information and search through content, and through content accessible using deep links in apps.

Apple Already has search. If you can use their search to help people find content that they are interested in, there’s value in being found using Apple search.

subscribe to our newsletter

2 Comments

  1. Andy

    September 09th, 2020 at 11:11 am

    Vipul Ved Prakash isn’t with Apple any more but Rishab Ghosh is.

    https://www.linkedin.com/in/rishab-ghosh-a17b2/

    He is on a lot of patents
    https://patents.justia.com/inventor/rishab-aiyer-ghosh

    The only internal contact I had at Topsy is now in digital banking

    Reply

    • Bill Slawski

      September 09th, 2020 at 11:54 am

      Thanks Andy,

      I started reading through patents from Ghosh last week. There is some interesting stuff there.

Leave a Comment