Ingest API

Site Search > App Settings > Data Management > Ingest API

The SearchStax Site Search solution’s Ingest API exposes the project’s /update endpoint. It lets you update Solr documents in your Site Search project. For perspective on the advantages of using the Ingest API, see our Data Ingestion for Site Search blog post.

The /update endpoint uses the “Read-Write” Search API credentials from the Site Search > App Settings > All APIs > Search & Indexing screen.

  

Contents:

  

Indexing JSON Documents

For indexing a single JSON document, pass the JSON document to the /update endpoint as shown below:

For apps using Token Authentication:

1
curl -H "Authorization: Token b70dacf3956e6e148393fbfREDACTED" -X POST -H 'Content-type:application/json' --data-binary '{"id":"1"}' "https://searchcloud-2-us-east-1.searchstax.com/12345/custom-1234/update?commit=true"

For app using Basic Authentication:

1
2
3
4
5
curl -X POST -H 'Content-Type: application/json' 'https://ss123456-cvrfzabx-us-east-1-aws.searchstax.com/solr/ss123456-SearchStudioCorpSite/update/json/docs' --user app40-admin:REDACTEDPWD --data-binary '
{
  "id": "1",
  "title": "Doc 1"
}'

Adding multiple documents can be done by passing an array of JSON objects to the /update endpoint as shown below

1
curl -H "Authorization: Token b70dacf3956e6e148393fbfREDACTED" -X POST -H 'Content-type:application/json' --data-binary '[{"id":"1","title_t":"Test Title","description_t":"Testing","group_s":"Token"}]' "https://searchcloud-2-us-east-1.searchstax.com/12345/custom-1234/update?commit=true"
1
2
3
4
5
6
7
8
9
10
11
curl -X POST -H 'Content-Type: application/json' 'https://ss123456-cvrfzabx-us-east-1-aws.searchstax.com/solr/ss123456-SearchStudioCorpSite/update' --user app40-admin:REDACTEDPWD --data-binary '
[
  {
    "id": "1",
    "title": "Doc 1"
  },
  {
    "id": "2",
    "title": "Doc 2"
  }
]'

If the data is in a JSON file and contains an array of JSON objects, you can make a call to the /update endpoint as shown below

1
curl -H "Authorization: Token b70dacf3956e6e148393fbfREDACTED" -X POST -H 'Content-type:application/json' --data-binary @example.json "https://searchcloud-2-us-east-1.searchstax.com/12345/custom-1234/update?commit=true"
1
curl -X POST -H 'Content-Type: application/json' 'https://ss123456-cvrfzabx-us-east-1-aws.searchstax.com/solr/ss123456-SearchStudioCorpSite/update' --user app40-admin:REDACTEDPWD --data-binary @example.json -H 'Content-type:application/json'
  

Indexing XML Documents

To index XML documents, you can send one or multiple <add> tags to the \update request as shown below:

For apps using Token Authentication:

1
2
3
4
5
6
7
8
9
curl -H "Authorization: Token b70dacf3956e6e148393fbfREDACTED" -X POST -H 'Content-type:text/xml' --data-binary
 '&lt;add>
    &lt;doc>
      &lt;field name="id">id1&lt;/field>
      &lt;field name="title">Document1&lt;/field>
      &lt;field name="text">This is the first document&lt;/field>
    &lt;/doc>
  &lt;/add>'
"https://searchcloud-2-us-east-1.searchstax.com/12345/custom-1234/update?commit=true"

For apps using Basic Authentication:

1
2
3
4
5
6
7
8
curl -X POST -H 'Content-Type: text/xml' 'https://ss123456-cvrfzabx-us-east-1-aws.searchstax.com/solr/ss123456-SearchStudioCorpSite/update?commit=true' --user app40-admin:REDACTEDPWD --data-binary '
&lt;add>
  &lt;doc>
    &lt;field name="id">id1&lt;/field>
    &lt;field name="title">Document1&lt;/field>
    &lt;field name="text">This is the first document&lt;/field>
   &lt;/doc>
&lt;/add>'
  • The <add> element introduces one more documents to be added. This can be substituted by <update> to update the document.
  • The <doc> element introduces the fields making up a document.
  • The <field> element presents the content for a specific field.
  

Indexing CSV Documents

The /update endpoint can load data files in comma-separated format (CSV). The format of the data file is very simple, as shown in the books.csv example that is distributed with Solr stand-alone installation files:

1
2
3
4
5
6
7
8
9
10
11
id,cat,name,price,inStock,author,series_t,sequence_i,genre_s
0553573403,book,A Game of Thrones,7.99,true,George R.R. Martin,"A Song of Ice and Fire",1,fantasy
0553579908,book,A Clash of Kings,7.99,true,George R.R. Martin,"A Song of Ice and Fire",2,fantasy
055357342X,book,A Storm of Swords,7.99,true,George R.R. Martin,"A Song of Ice and Fire",3,fantasy
0553293354,book,Foundation,7.99,true,Isaac Asimov,Foundation Novels,1,scifi
0812521390,book,The Black Company,6.99,false,Glen Cook,The Chronicles of The Black Company,1,fantasy
0812550706,book,Ender's Game,6.99,true,Orson Scott Card,Ender,1,scifi
0441385532,book,Jhereg,7.95,false,Steven Brust,Vlad Taltos,1,fantasy
0380014300,book,Nine Princes In Amber,6.99,true,Roger Zelazny,the Chronicles of Amber,1,fantasy
0805080481,book,The Book of Three,5.99,true,Lloyd Alexander,The Chronicles of Prydain,1,fantasy
080508049X,book,The Black Cauldron,5.99,true,Lloyd Alexander,The Chronicles of Prydain,2,fantasy
  • Line 1 is the header, containing the field names used in the index. One of these must be a unique document ID.
  • Subsequent lines each represent one index record. Commas separate values. Do not allow spaces immediately following a comma.
  • A missing value is denoted by two adjacent commas (,,).
  • Note the dynamic fields (series_t, sequence_i, genre_s) in the header. This mechanism lets you add custom fields to your index without the labor of modifying the Solr schema.

For apps using Token Authentication:

1
2
3
4
curl -H "Authorization: Token 8a931ae9bfe3538c2b4d8121fad564c676f7de0f"
  -H 'Content-type:application/csv'
  --data-binary @books.csv
  'https://searchcloud-1-us-west-2.searchstax.com/95338/csvuploadapp-3684/update?commit=true

For apps using Basic Authentication:

1
2
3
4
curl -H 'Content-type:application/csv'
  --data-binary @books.csv
  --user app40-admin:REDACTEDPWD
  'https://searchcloud-1-us-west-2.searchstax.com/95338/csvuploadapp-3684/update?commit=true

Deleting Documents

When experimenting with document uploads, it is often helpful to be able to clear the index or to remove specific documents. See the following Help Center topics for examples:

  

API Limits

  • All update requests should have a maximum size of 2048KB.

Questions?

Do not hesitate to contact the SearchStax Support Desk.