This page is about writing software to send data from a factory to Instrumental. It assumes that you have read How do I import and use functional test data? and have worked with an Instrumental Solutions Architect to decide that this is the best implementation option. If that’s the case, that means you have a way to obtain the data you want to send, as well as a place you can run the integration script you will need to write.
After you purchase Data Streams, Instrumental will set up a test project in the Instrumental Web App so that you can develop an integration without sending test data to your “real” project.
Table of Contents
API Keys
To send data, you will need an API Key with “write” permissions. You can read about API Keys and how to obtain them on the API Keys page.
Data Streams API
Domain/endpoint
Send data to https://api.instrumental.ai/api/v1/externalData/ingest with the POST HTTP method.
Please work with your networking teams to unblock TCP on port 443 (HTTPS access) for this domain on whatever network your integration script will run. The domain does not have a fixed IP address or IP range, so the domain itself must be allowed. If necessary, Instrumental can discuss setting up a proxy with a fixed IP.
Requests must have the following headers:
instrumental-api-key: YOUR_API_KEY
content-type: application/json
Of course, replace YOUR_API_KEY with your actual, full key (it should contain INST:V1 – don’t use only the identifier displayed in the API key modal).
Default limits
Ingest requests may be up to 4MB. Larger requests will be rejected with a 413 Too Large error.
Clients have a rate limit of 50 ingest requests refilled at a rate of 5 per second, shared across all ingest API endpoints. This allows some burst capacity if you need to send more requests for a short time. In some circumstances rate limits may be applied per factory site. If the rate limit is exceeded, requests will be rejected with a 429 Too Many Requests error until the rate limit bucket refills enough for new requests to get through.
Additionally, there is a separate rate limit on measurement limit updates. To prevent a situation where measurement limits are changed constantly, creating a long history, the limits for each measurement can only be changed to a different value by the API at most once every 6 hours. Measurement limit updates that are more frequent than that, and those that do not change the limits, will be ignored (they will not affect whether the rest of the request is successful).
Each unit (unique serial number) may have no more than 100 Data Streams inspections and 10,000 total Data Streams measurements. A merged unit may have no more than 349 subassemblies. To avoid hitting these limits, please review the Data Structure Recommendations section below. In cases where subassemblies are merged into a final assembly, these limits apply to the merged unit, not to each component individually. If a merged unit exceeds these limits, only a subset of data will be shown, along with a warning that this is the case.
Each ingest request may include up to 100 inspections.
If you expect to exceed any of the limits described above, please discuss with your Account Manager or Solutions Architect.
Request
The endpoint accepts a JSON body with this structure:
{
"inspections": [
{
"serialNumber": "SERIAL_NUMBER_123",
"stationName": "Station name",
"timestamp": {
"iso8601Time": "YYYY-MM-DDTHH:mm:ss.SSSZ",
"ianaTimeZone": "Asia/Shanghai"
},
"data": [
{
"name": "Measurement name",
"value": {
// Choose (only) one of the following properties:
"string": "Measurement value",
"double": -1.0,
"boolean": true,
"in.datatypes.TimeWithZone": {
"iso8601Time": "2021-02-18T14:31:07.469-08:00",
"ianaTimeZone": "America/Los_Angeles"
}
},
"children": [
...
],
},
...
],
// Optional; you can omit "measurementLimits" if the measurements in this inspection have no associated limits
"measurementLimits": {
// Ensure that the name actually matches and the limit name is not e.g. "Measurement name limit"
"Measurement name": {
// Choose (only) one of the following properties:
"in.datatypes.measurementlimit.StringMeasurementLimit": {
// If there are no keywords for one of the fields, provide an empty array
"passingKeywords": ["PASS", "success"],
"failingKeywords": ["FAIL"]
},
"in.datatypes.measurementlimit.BooleanMeasurementLimit": {
"trueIsPassing": true
},
"in.datatypes.measurementlimit.NumberMeasurementLimit": {
// To provide only the upper or lower limit, omit the unused one
"passingUpperBound": 100,
"passingLowerBound": 0
},
"in.datatypes.measurementlimit.DateTimeMeasurementLimit": {
// To provide only the upper or lower limit, omit the unused one
"passingUpperBound": {
"iso8601Time": "YYYY-MM-DDTHH:mm:ss.SSSZ",
"ianaTimeZone": "Asia/Shanghai"
},
"passingLowerBound": {
"iso8601Time": "YYYY-MM-DDTHH:mm:ss.SSSZ",
"ianaTimeZone": "Asia/Shanghai"
}
}
},
...
},
// Optional; you can omit "parentAssembly" if this unit has none
"parentAssembly": {
"unitSerial": "Parent SN",
// The name of the assembly being uploaded (not the name of the parent assembly)
"relationshipName": "Display"
},
// Optional; you can omit "subassemblies" if this unit has none
"subassemblies": [
{
"unitSerial": "Child SN",
// The name of the subassembly
"relationshipName": "Coverglass"
},
...
]
},
...
],
// Optional; you can omit "unitConfigs"
"unitConfigs": {
"SERIAL_NUMBER_123": "Config name",
}
}serialNumber: string
The serial number associated with the inspected unit. All test data will be tied to this serial number. If you have test data that is not associated with a specific serial number (e.g. test data for a lot code but not specific units in that lot) please consult with your Instrumental Account Manager about what to put in this field.
stationName: string
Name of the test station where the data was measured.
timestamp.iso8601Time: ISO 8601 datetime string
The time at which this measurement was taken, as an ISO 8601 time string, i.e.: “YYYY-MM-DDTHH:mm:ss.SSSZ”. The string does not have to be in UTC, but note that any offset here will be used only for determining the moment in time — not for any additional time zone math (e.g. calculating the “clock-on-the-wall” time). Instead, the zone string provided in the “ianaTimeZone” field of this record will be used for that.
timestamp.ianaTimeZone: DateTimeZone
The time zone in which this measurement was taken, as a time zone name found in the IANA tz database, e.g.: “America/Los_Angeles”, “Asia/Shanghai”.
data: array[APIExternalData]
Test data or metadata associated with the given serial number. The APIExternalData structure is defined below. If the array is empty, no data will be saved, but this is still treated as valid so that there is a way to send measurementLimits, parentAssembly, and subassemblies without adding inspections.
APIExternalData
name: string
The name of the measurement, e.g. Station 1 Test Result. Must not be the empty string. Note that names should be unique across stations. For example, prefer names like Station 1 Duration rather than simply Duration. (For best usability, aim to keep names under 1000 characters.) See also the Data Type Recommendations section below.
value: { string } or { double } or { boolean } or timestamp
The value associated with the measurement. This is represented as a JSON object with one key, the type, and its associated value. The type can either be string, boolean, double, or in.datatypes.TimeWithZone:
string– indicates the value is arbitrary textboolean– a true/false value, for example representing a pass/fail test resultdouble– accepts any signed decimal number in the range -253 to +253-1 (15 significant digits)- timestamp – a datetime with a time zone. As above, this is represented as a JSON object with properties
iso8601Time: DateTimeandianaTimeZone: DateTimeZone.
The data type you choose is important! It affects how data appears in charts and correlations. For example, avoid sending numbers as strings. See also the Data Type Recommendations section below.
children: array[APIExternalData]?
This optional property allows you to nest data, creating a hierarchy. You can further nest more data under the children data objects. The maximum nesting depth of child data is 30.
measurementLimits: map[string, Limit]?
An optional map from measurement names to the associated limits. Be careful that the limit names (the keys of the map) actually match measurement names (values that show up in the "name" field for APIExternalData entries). For example, if you have a measurement for “WiFi RSSI”, the limit name should also be “WiFi RSSI” and not “WiFi RSSI limit” or “WiFi RSSI upper limit”. (It is valid, though unusual, for individual inspections to include limits whose name does not match the name of a measurement in that inspection’s "data" array. This is allowed so that it’s possible to send limits separately from the data. However, in that case you should still make sure the limit names are consistent with the separately uploaded measurements.)
The map’s values are a data structure that defines limits for each supported data type.
Note that users can “lock” limits in the Instrumental web app, preventing manual changes from being overridden via the API. Also, as noted in the Default Limits section above, updating measurement limits has a fairly restrictive rate limit.
parentAssembly: APIExternalDataAssemblyRelationship?
subassemblies: array[APIExternalDataAssemblyRelationship]?
Note: these fields are only available to projects created after April 18, 2022. Contact Instrumental if you want to use them for projects created before that.
These fields define how components with their own serial numbers are combined to produce the final assembly. For example, a device might have a structure like this:
Final assembly: ENC2124
├─ PCB subassembly: PCB4149
├─ Display subassembly: DIS8801
| ├─ Coverglass subassembly: GLS1035
├─ Battery subassembly: BAT7257It is likely that the relationships will not be known at every inspection step – for example, when the battery is inspected, it may not be known which enclosure it will go into. That’s fine; the relationships only have to be uploaded once, and only in one direction (though doing it again won’t hurt).
Assemblies can have multiple subassemblies but only one “active” parent. If an assembly already has a parent and you upload a relationship that would give it a different parent, the more recent request becomes the “active” one. Please note that assembly relationships may not have cycles. That is, an error will be returned when attempting to upload a relationship that would cause an assembly to have another assembly among both its ancestors and descendants – even if one of the links in that chain is not “active.”
If you upload a subassembly without a parent, it will initially be represented as its own unit in the Instrumental app. When a relationship with a parent is established, it will be merged with the parent unit. This merging allows data for all the components in a final assembly to be reviewed and correlated together. If a component’s active parent assembly changes, it will be un-merged from its previous parent and merged into the new one.
If you are doing a bulk upload, your data will be available in the app faster if you upload units’ relationships before or in the same request as you upload those units’ first inspections because Instrumental can then merge the assemblies immediately instead of creating them separately and merging them later.
APIExternalDataAssemblyRelationship
unitSerial: string
The serial number of the unit on the other side of the described relationship.
relationshipName: string
The name of the subassembly as it relates to the parent assembly, e.g. “Left display”.
unitConfigs: map[string, string]
A map from serial numbers to unit config names. Each unit may have only one unit config (note this is different than “image configs” which apply to individual photos). If a unit specified here already has a unit config, its unit config will be overwritten.
Example request body
{
"inspections": [
{
"serialNumber": "123456789ABC",
"stationName": "Functional Test Station",
"timestamp": {
"iso8601Time": "2020-04-09T14:52:10.499Z",
"ianaTimeZone": "America/Los_Angeles"
},
"data": [
{
"name": "PRODUCT NAME",
"value": {
"string": "Product Name"
},
"children": [
{
"name": "GROUP NAME",
"value": {
"string": "Group 1"
}
},
{
"name": "STEPGROUP",
"value": {
"string": "Main"
}
},
{
"name": "GROUPINDEX",
"value": {
"double": 443.0
}
},
{
"name": "LOOPINDEX",
"value": {
"double": -1.0
}
},
{
"name": "TYPE",
"value": {
"string": "PassFailTest"
}
},
{
"name": "RESOURCE",
"value": {
"string": ""
}
},
{
"name": "MODULETIME",
"value": {
"double": 0.351136
}
},
{
"name": "TOTALTIME",
"value": {
"double": 0.351303
}
},
{
"name": "TIMESTAMP",
"value": {
"in.datatypes.TimeWithZone": {
"iso8601Time": "2020-02-14T12:59:31.469+08:00",
"ianaTimeZone": "America/Los_Angeles"
}
}
},
{
"name": "STATUS",
"value": {
"string": "Passed"
}
},
{
"name": "Sample_boolean",
"value": {
"boolean": true
}
}
]
}
],
"measurementLimits": {
"GROUPINDEX": {
"in.datatypes.measurementlimit.NumberMeasurementLimit": {
"passingUpperBound": 500,
"passingLowerBound": 400
}
},
"STATUS": {
"in.datatypes.measurementlimit.StringMeasurementLimit": {
"passingKeywords": ["Passed"],
"failingKeywords": ["Failed"]
}
}
}
}
]
}Here is that request via cURL:
curl -v -XPOST --data '{"inspections":[{"serialNumber":"123456789ABC","stationName":"FunctionalTestStation","timestamp":{"iso8601Time":"2020-04-09T14:52:10.499Z","ianaTimeZone":"America/Los_Angeles"},"data":[{"name":"PRODUCTNAME","value":{"string":"ProductName"},"children":[{"name":"GROUPNAME","value":{"string":"Group1"}},{"name":"STEPGROUP","value":{"string":"Main"}},{"name":"GROUPINDEX","value":{"double":443.0}},{"name":"LOOPINDEX","value":{"double":-1.0}},{"name":"TYPE","value":{"string":"PassFailTest"}},{"name":"RESOURCE","value":{"string":""}},{"name":"MODULETIME","value":{"double":0.351136}},{"name":"TOTALTIME","value":{"double":0.351303}},{"name":"TIMESTAMP","value":{"in.datatypes.TimeWithZone":{"iso8601Time":"2020-02-14T12:59:31.469+08:00","ianaTimeZone":"America/Los_Angeles"}}},{"name":"STATUS","value":{"string":"Passed"}},{"name":"Sample_boolean","value":{"boolean":true}}]}],"measurementLimits":{"GROUPINDEX":{"in.datatypes.measurementlimit.NumberMeasurementLimit":{"passingUpperBound":500,"passingLowerBound":400}},"STATUS":{"in.datatypes.measurementlimit.StringMeasurementLimit":{"passingKeywords":["Passed"],"failingKeywords":["Failed"]}}}}]}' -H 'instrumental-api-key: YOUR_API_KEY' -H 'content-type: application/json' https://api.instrumental.ai/api/v1/externalData/ingest(Make sure to replace YOUR_API_KEY with your actual API key if you want to test this.)
Response
If the request is successful, the response will have an HTTP status code of 200. Failed requests will receive one of the following HTTP status codes:
- 400 – Failure to parse the input body
- 401 – Invalid API key in the header
- 413 – Request is too large; see the Default limits section
- 429 – Rate limits exceeded; see the Default limits section
- 500 – Server error
If a request fails with one of these status codes, none of the parts of the request will be saved.
The response body for a successful request (status code 200) will contain a JSON representation of the results of the request that looks like this:
{
"results": [
{
// Only one of these fields will be present:
"success": ...,
"failure": ...,
},
...
]
}Each entry in the "results" array corresponds to the inspection at the same index in the "inspections" array in the request. It will have either a "success" or a "failure" field indicating whether the inspection was saved successfully. In the success case, the value of the field will be a representation of the saved inspection. In the error case, the failure field will be an object containing an error code and a human-readable msg explaining what went wrong.
As you’ll see below, successful results have an inspection key with details about the saved unit/inspection/measurements. They also have a measurementLimitResults key which holds a map from specified limit names to results for saving them. Just like at the higher level, these measurement limit results are objects with either a success or failure key indicating the outcome. Measurement limits can fail to save due to rate limiting, being identical to existing limits, or a user locking updates through the web app. In each case, only the measurement limit will fail; inspection data can still save successfully and the returned HTTP response code can be 200.
If an inspection is submitted with exactly the same API key, serialNumber, stationName, timestamp, and data as a previous request, it is considered a duplicate and is not saved. (“Exactly the same data” includes the order of the data, which may not be obvious once submitted because the data can be sorted differently when displayed in the UI.) This will be treated as a "failure" in the response but the HTTP status code will still be 200. measurementLimits are not used to check if an inspection is a duplicate, and can still be saved if the rest of the inspection is a duplicate.
If the whole request cannot be processed, the response will explain why. For example:
Bad input:
{
"inspections": [
{
"serialNumber": "123456789ABC",
"stationName": "Functional Test Station",
"timestamp": {
"iso8601Time": "2020-04-09T14:52:10.499Z",
"ianaTimeZone": "America/Los_Angeles"
},
"data": [
{
"name": "PRODUCT NAME",
"value": {
"string": "Product Name"
},
"children": [
{
"name": "GROUP NAME",
"value": {
"string": "gpio set_level a 10 1"
}
},
{
"name": "GROUPINDEX",
"value": {
"double": "443.0" // <-- error here: value has type string
}
}
]
}
]
}
]
}Will return 400 Bad Request:
{
"error": "Could not parse request body as the expected JSON format.",
"reason": "ERROR :: /inspections/0/data/0/children/1/value/double :: sending a string instead cannot be coerced to Double"
}Next is an example request to upload the same inspection twice. Note the limits are different, but otherwise the second inspection is identical to the first, so the second entry in the response has a failure code identifying it as a duplicate. (The request will still have an HTTP status code of 200.) This example illustrates the response structure you can expect from successful requests.
Good input:
{
"inspections": [
{
"serialNumber": "SN20",
"stationName": "Test station 1",
"timestamp": {
"iso8601Time": "2020-01-01T08:00:00.000+08:00",
"ianaTimeZone": "Asia/Shanghai"
},
"data": [
{
"name": "string data",
"value": {
"string": "string data 20"
}
},
{
"name": "number data",
"value": {
"double": 20
}
},
{
"name": "boolean data",
"value": {
"boolean": true
}
},
{
"name": "date data",
"value": {
"in.datatypes.TimeWithZone": {
"iso8601Time": "2020-01-01T08:00:00.000+08:00",
"ianaTimeZone": "Asia/Shanghai"
}
}
}
],
"measurementLimits": {
"number data": {
"in.datatypes.measurementlimit.NumberMeasurementLimit": {
"passingLowerBound": 0
}
},
"boolean data": {
"in.datatypes.measurementlimit.BooleanMeasurementLimit": {
"trueIsPassing": true
}
}
}
},
{
"serialNumber": "SN20",
"stationName": "Test station 1",
"timestamp": {
"iso8601Time": "2020-01-01T08:00:00.000+08:00",
"ianaTimeZone": "Asia/Shanghai"
},
"data": [
{
"name": "string data",
"value": {
"string": "string data 20"
}
},
{
"name": "number data",
"value": {
"double": 20
}
},
{
"name": "boolean data",
"value": {
"boolean": true
}
},
{
"name": "date data",
"value": {
"in.datatypes.TimeWithZone": {
"iso8601Time": "2020-01-01T08:00:00.000+08:00",
"ianaTimeZone": "Asia/Shanghai"
}
}
}
],
"measurementLimits": {
"number data": {
"in.datatypes.measurementlimit.NumberMeasurementLimit": {
"passingUpperBound": 100
}
}
}
}
]
}Response:
{
"results": [
{
"success": {
"inspection": {
"serialNumber": "SN20",
"stationName": "Test station 1",
"timestamp": {
"ianaTimeZone": "Asia/Shanghai",
"iso8601Time": "2020-01-01T08:00:00.000+08:00"
},
"measurements": [
{
"name": "string data",
"id": "wo8zbqyvbqO6",
"value": {
"string": "string data 20"
}
},
{
"name": "number data",
"id": "6vRVbxMm5qra",
"value": {
"double": 20
}
},
{
"name": "boolean data",
"id": "qg245pNNLeZz",
"value": {
"boolean": true
}
},
{
"name": "date data",
"id": "DYe0BldABQ2j",
"value": {
"in.datatypes.TimeWithZone": {
"ianaTimeZone": "Asia/Shanghai",
"iso8601Time": "2020-01-01T08:00:00.000+08:00"
}
}
}
]
},
"measurementLimitResults": {
"boolean data": {
"success": {
"limitValue": {
"in.datatypes.measurementlimit.BooleanMeasurementLimit": {
"trueIsPassing": true
}
},
"createdAt": "2021-05-17T22:37:22.191Z",
"id": "OY9Z56KB1o42",
"measurementName": "boolean data",
"updatedAt": "2021-05-17T22:37:22.196Z"
}
},
"number data": {
"success": {
"limitValue": {
"in.datatypes.measurementlimit.NumberMeasurementLimit": {
"passingLowerBound": 0
}
},
"createdAt": "2021-05-17T22:37:22.190Z",
"id": "NRo8bnrbvkQe",
"measurementName": "number data",
"updatedAt": "2021-05-17T22:37:22.196Z"
}
}
}
}
},
{
"failure": {
"msg": "An inspection with the same hash was previously uploaded with the same API key.",
"code": "DUPLICATE_INSPECTION"
}
}
]
}If the request is not formatted as valid JSON or if the JSON structure does not match the format described above, the response body will have a different format that describes what is wrong. For example, if you try to send a string as the value of a measurement instead of using the { "string": "value" } format, you may see a message like this:
{"error":"Could not parse request body as the expected JSON format.","reason":"ERROR :: /inspections/0/data/0/value :: union type is not backed by a DataMap or null"}Reviewing errors
API requests that authenticate successfully (i.e. they have a valid instrumental-api-key header and are not rate limited) but fail other validation (e.g. the request body is invalid) will be temporarily accessible for debugging through the API key modal. You can read more about this on the API Keys page.
Data Structure Recommendations
Test/metric names
Do not reuse a name to describe a data point if the values are not comparable. For example, a common mistake is to have a test named “Result” at multiple stations. Instead, prepend the station name; for example, at an RF station, the name for the overall pass/fail output could be “RF Result.” (It is okay if the resulting name is long. For best usability, aim to keep names under 1000 characters.)
Keep in mind that the name field for each measurement in the data array should identify the measurement values it is paired with. For example, if a test station produces a log like this:
Test name: Charging Current Test value: 841.27 Test name: Battery Voltage Test value: 3.813
Your data array should look like this:
[
{
"name": "Charging Current",
"value": {
"double": 841.27
}
},
{
"name": "Battery Voltage",
"value": {
"double": 3.813
}
}
]and not like this:
[
{
"name": "Test name",
"value": {
"string": "Charging Current"
}
},
{
"name": "Test value",
"value": {
"double": 841.27
}
},
{
"name": "Test name",
"value": {
"string": "Battery Voltage"
}
},
{
"name": "Test value",
"value": {
"double": 3.813
}
}
]The latter is problematic because it mixes together different kinds of values. The former is better because it allows search filters like “Charging Current < 850” as well as more intuitive usage with exploring data histograms and correlations in Instrumental’s web app.
Along similar lines, each inspection should only have one test with each test name. If you run a test multiple times you should either treat that as separate inspections or give the tests different names. For example, if you run a power check at the beginning and end of a test, call them “Initial Power Check” and “Final Power Check” rather than calling both checks “Power Check.” Otherwise, charts and statistics may conflate the two values and you won’t be able to use “first pass”/”last pass” filters to separate them out.
Test/metric values
Ensure that your data is sent using the correct data type. The most common mistake is to send numbers and dates as strings.
Avoid sending data with different data types under the same test name. For example, a common mistake is to send the values N/A, Null, NaN, #NUM!, an empty string, etc. as string measurements for a test that normally outputs numbers. Instead, those “null values” should be omitted from the upload. (It is okay if not every inspection has the same tests.)
Sweeps
Frequency sweeps are often represented in raw data (e.g. CSVs) as hundreds or thousands of measurements per unit (e.g. columns in a CSV). Instrumental is developing a more compact way to upload sweep data. In the meantime, consider whether you can limit the upload to just the most useful frequencies to avoid hitting the limit on the maximum amount of data allowed per unit.
Reliability Tests
Reliability tests are often represented in raw data (e.g. CSVs) as hundreds of inspections per unit, each with dozens or hundreds of measurements. (In a CSV, each row typically represents an inspection and each cell typically represents a measurement.) However, attempting to ingest this many inspections for a single unit into Instrumental will likely exceed the allowed limits (and even if not, it would be difficult to look through in the UI). Instead, consider whether you can send a summary, e.g. an overall Pass/Fail result and specific measurements that failed, as a single inspection per unit.
Test limits
Instrumental supports specifically identifying test limits in the request data structure, as described above. Be careful to send limits in the measurementLimits field, rather than as measurement values in the data array, to avoid unnecessarily doubling or tripling your measurement count and request size.
Ensure that test limit names exactly match the names of the tests the limits are for. Also ensure that the limits have the same data type as the corresponding test data. For example, string limits for numeric data will be ignored.
You may want to infer “limits” from your data even if the source data does not specifically identify limits. For example, if you have a test whose output is “PASS” or “FAIL”, consider adding a limit to identify “PASS” as passing and “FAIL” as failing. Similarly, Boolean true/false values may be a good candidate for inferred limits. If you do not specify limits, Instrumental will not add them for you, although heuristics will sometimes be used to improve the experience and you can manually update them in the UI later.
Examples
For the following CSV:
Serial,Timestamp,Station Name,Overall Result,Battery Voltage [V],Battery Voltage Min [V],Battery Voltage Max [V],Test Suite Version
RY76253,2019-10-05T07:22:10.000Z,Final Functional Test,TRUE,3.873,3.75,3.9,1.2a_DVT
RY35726,2019-10-05T08:01:07.000Z,Final Functional Test,FALSE,3.992,3.75,3.9,1.2a_DVT
RY73014,2019-10-05T07:23:36.000Z,Final Functional Test,TRUE,3.808,3.75,3.9,1.2a_DVT
This Python program can parse and upload it if you have the requests library installed:
#!/usr/bin/env python3
import argparse
import csv
import json
import datetime
import pdb
import re
import requests
children = {
# Use this to map columns into a hierarchy if needed, e.g. numeric tests that result in a summary pass/fail status
# Example: "BLE_2412MHz_RSSI [dBm]": "BLE_Antenna Test [P/F]",
}
parents = set(children.values())
# Identify columns we do not want to upload by their name in the header row
skip_cols = { "Program", "Some other header" }
# Identify columns containing subassembly serial numbers
subassembly_cols = { "Left display subassembly", "Right display subassembly" }
def add_implicit_limits_if_needed(meas, limits):
""" Adds default limits for values that look like they're clearly passing or failing """
if "string" in meas["value"] and re.search(r'^(pass|fail)$', meas["value"]["string"], re.IGNORECASE) and not meas["name"] in limits:
limits[meas["name"]] = {
"in.datatypes.measurementlimit.StringMeasurementLimit": {
"passingKeywords": ["PASS"],
"failingKeywords": ["FAIL"],
},
}
elif "boolean" in meas["value"] and not meas["name"] in limits:
limits[meas["name"]] = {
"in.datatypes.measurementlimit.BooleanMeasurementLimit": {
"trueIsPassing": True,
},
}
def get_value(val):
""" Returns the value in the specified CSV cell with the appropriate data type """
if val.lower() == "true" or val.lower() == "false":
return { "boolean": val.lower() == "true" }
else:
try:
return { "double": float(val) }
except:
return { "string": str(val) }
def build_row(csv_row):
""" Creates an Instrumental Data Streams inspection request object for each CSV row """
edi_dict = {
"serialNumber": csv_row.pop("Serial"),
"stationName": csv_row.pop("Station Name"),
"timestamp": {
"iso8601Time": datetime.datetime.strptime(csv_row.pop("Timestamp"), "%Y-%m-%dT%H:%M:%S.%fZ").isoformat(timespec="milliseconds") + "Z",
"ianaTimeZone": "America/Los_Angeles",
},
"data": [],
"measurementLimits": {},
"subassemblies": []
}
parent_data = {}
for parent in parents:
parent_data[parent] = {
"name": parent,
"value": get_value(csv_row.pop(parent)),
"children": [],
}
edi_dict["data"].append(parent_data[parent])
add_implicit_limits_if_needed(parent_data[parent], edi_dict["measurementLimits"])
for test in csv_row:
if test in skip_cols:
continue
elif csv_row[test] == "":
continue
elif test in subassembly_cols:
edi_dict["subassemblies"].append({ "unitSerial": csv_row[test], "relationshipName": test })
elif re.search(r'\s(lower limit|LSL)$', test, re.IGNORECASE):
limit_name = re.sub(r'\s(lower limit|LSL)$', "", test, flags=re.IGNORECASE)
if limit_name in edi_dict["measurementLimits"]:
edi_dict["measurementLimits"][limit_name]["in.datatypes.measurementlimit.NumberMeasurementLimit"]["passingLowerBound"] = float(csv_row[test])
else:
edi_dict["measurementLimits"][limit_name] = {
"in.datatypes.measurementlimit.NumberMeasurementLimit": {
"passingLowerBound": float(csv_row[test]),
},
}
elif re.search(r'\s(upper limit|USL)$', test, re.IGNORECASE):
limit_name = re.sub(r'\s(upper limit|USL)$', "", test, flags=re.IGNORECASE)
if limit_name in edi_dict["measurementLimits"]:
edi_dict["measurementLimits"][limit_name]["in.datatypes.measurementlimit.NumberMeasurementLimit"]["passingUpperBound"] = float(csv_row[test])
else:
edi_dict["measurementLimits"][limit_name] = {
"in.datatypes.measurementlimit.NumberMeasurementLimit": {
"passingUpperBound": float(csv_row[test]),
},
}
elif re.search(r'\spassing value$', test, re.IGNORECASE):
limit_name = re.sub(r'\spassing value$', "", test, flags=re.IGNORECASE)
if limit_name in edi_dict["measurementLimits"]:
edi_dict["measurementLimits"][limit_name]["in.datatypes.measurementlimit.BooleanMeasurementLimit"]["trueIsPassing"] = (csv_row[test] == "true")
else:
edi_dict["measurementLimits"][limit_name] = {
"in.datatypes.measurementlimit.BooleanMeasurementLimit": {
"trueIsPassing": csv_row[test] == "true",
},
}
elif re.search(r'\sfailing value$', test, re.IGNORECASE):
limit_name = re.sub(r'\sfailing value$', "", test, flags=re.IGNORECASE)
if limit_name in edi_dict["measurementLimits"]:
edi_dict["measurementLimits"][limit_name]["in.datatypes.measurementlimit.BooleanMeasurementLimit"]["trueIsPassing"] = (csv_row[test] != "true")
else:
edi_dict["measurementLimits"][limit_name] = {
"in.datatypes.measurementlimit.BooleanMeasurementLimit": {
"trueIsPassing": csv_row[test] != "true",
},
}
elif re.search(r'\spassing values$', test, re.IGNORECASE):
limit_name = re.sub(r'\spassing values$', "", test, flags=re.IGNORECASE)
if limit_name in edi_dict["measurementLimits"]:
edi_dict["measurementLimits"][limit_name]["in.datatypes.measurementlimit.StringMeasurementLimit"]["passingKeywords"] = csv_row[test].split(';')
else:
edi_dict["measurementLimits"][limit_name] = {
"in.datatypes.measurementlimit.StringMeasurementLimit": {
"passingKeywords": csv_row[test].split(';'),
},
}
elif re.search(r'\sfailing values$', test, re.IGNORECASE):
limit_name = re.sub(r'\sfailing values$', "", test, flags=re.IGNORECASE)
if limit_name in edi_dict["measurementLimits"]:
edi_dict["measurementLimits"][limit_name]["in.datatypes.measurementlimit.StringMeasurementLimit"]["failingKeywords"] = csv_row[test].split(';')
else:
edi_dict["measurementLimits"][limit_name] = {
"in.datatypes.measurementlimit.StringMeasurementLimit": {
"failingKeywords": csv_row[test].split(';'),
},
}
else:
row_data = {
"name": test,
"value": get_value(csv_row[test]),
}
if test in children:
parent_data[children[test]]["children"].append(row_data)
else:
edi_dict["data"].append(row_data)
add_implicit_limits_if_needed(row_data, edi_dict["measurementLimits"])
return edi_dict
def main():
""" Runs when the program starts. Try `python3 uploader.py --help` for info. """
parser = argparse.ArgumentParser(description="Upload Data Streams data from a CSV file")
parser.add_argument("infile", type=argparse.FileType("r"), help="the file to parse")
parser.add_argument("--apikey", type=str, default="", help="the project API key to use to upload")
parser.add_argument("--debug", action="store_true", help="pretty-print the request JSON instead of uploading")
parser.add_argument("--verbose", "-v", action="store_true", help="print request and response objects")
args = parser.parse_args()
if not args.debug and not args.apikey:
print("You must specify either an API key or the --debug flag")
return
row_count = sum(1 for _ in args.infile) - 1 # Subtract one for header row
args.infile.seek(0)
datareader = csv.DictReader(args.infile, delimiter=",")
url = "https://api.instrumental.ai/api/v1/externalData/ingest"
headers = {
"content-type": "application/json",
"instrumental-api-key": args.apikey,
}
current_request = { "inspections": [] }
if args.apikey: print("Starting upload for {} inspections...\n".format(row_count))
i = 0
for row in datareader:
i += 1
if args.debug and args.apikey: print("{}...".format(i))
built_row = build_row(row)
current_request["inspections"].append(built_row)
if i % 100 == 0 or i == row_count:
if args.apikey:
json_data = json.dumps(current_request)
print("Uploading {} inspection(s)...".format(len(current_request["inspections"])))
if args.verbose: print("REQUEST: {}".format(json_data))
response = requests.post(url, data=json_data, headers=headers)
if args.verbose: print("RESPONSE: {}".format(response.text))
if response.status_code != 200: pdb.set_trace()
else:
print(json.dumps(current_request, indent=4))
current_request = { "inspections": [] }
if args.apikey: print("Done!\n")
if __name__ == "__main__":
main()You can use this script as a starting point for writing your own integrations, or understanding how you can parse things like limits and subassemblies. You will most likely need to change it to accommodate your column names, timestamp format, time zone, etc. The built-in --debug and --verbose flags are helpful to validate that the requests (and responses) are as you expect.
Note that this program uses pdb, the Python debugger, to help you investigate request failures while writing the script. In a script you deploy and run unsupervised, you should instead log failures somewhere that allows you to retrieve them for debugging later.
API versions
Instrumental’s APIs may evolve over time, but we strive to keep any changes backwards-compatible. Below, we list some especially notable changes.
- May 2022: API keys now require “write” permissions to upload data. All keys created before this change already have “write” permissions.
- April 2022: The
subassembliesandparentAssemblyfields are introduced. These only work for projects created after this date. - March 2021: A new URL path is introduced:
/api/v1/externalData/ingest. The old/api/v0/externalData/ingestpath that only accepts one inspection at a time will continue to work.