Generate Demo data

The feature is to generate daily demo data to serve as an example platform for customers.

There are three ways to inject demo data:

1. Using the /api/playground url

2. and also having the daily demo data run automatically

3. run a rake task: rake demo:demodata[5] , to say insert 5 test results for each device template.

Setup

• enable demo data For automated tests you need to add to the docker-dev.env file: USE_DEMO_DATA=true

Note: for testing locally, to the config/settings.yml add: use_demo_data=true

• seed data For automated data generation [not needed for the playground but useful if done] it is necessary to generate the user/device/model/manifest’s in the database. This is done by firstly enabling demo data as above. this either: rake db:setup [which calls the seeds.rb] or rake db:seed

step 1: To generate demo data with the playground

Note: this only works on Chrome due to the DATETIME html5 element [we can replace it with a query plugin if needed].

Visit the url [local url is] http://localhost:3000/api/playground

• select a device created by the seed data such as Cepheid,

• for the data paste in a demo template in seeds/manifests_demo_template, for example for the cepheid device use cepheid template.

select number of test results to generate:

try setting a date [also if you do not set a start/end date it defaults to start today, and end today +24 hours if no date set]

To verify the results in mysql: select * from samples order by id desc; it also writes to: select count(*) from samples;

Note: look at the start_time, end_time fields inside the test_results to verify the dates generated.

Note: A useful command when stopping/starting to test locally is to clear the redis of jobs: redis-cli flushall

Note: There is a maximum limit of 10,000 in the code for the number of test results that can be generated.

To generate a single test result then enter:

centipede device:

{
  "event": {
    "result": "MTB DETECTED MEDIUM",
    "assay_name": "MTB-RIF Assay G4_noQC1_2",
    "sample_ID": "A-1234-5",
    "system_user": "JaneDoe",
    "device_serial_number": "123456",
    "end_time": "2015-04-07T20:08:41-05:00",
    "start_time": "2015-04-07T18:31:20-05:00",
    "event_id": "76B4AFBA09A143718CED4767ACC4EAE7",
    "test_type": "0"
  }
}

But to generate multi test results then use this syntax below, the {..} will be replaced with random data if the backend supports it:

{
  "event": {
    "result": "{result}",
    "assay_name": "MTB-RIF Assay G4_noQC1_2",
    "sample_ID": "{sample_id}",
    "system_user": "{system_user}",
    "device_serial_number": "123456",
    "end_time": "{end_datetime}",
    "start_time": "{start_datetime}",
    "event_id": "{event_id}",
    "test_type": "{test_type}"
  }
}

For the Fio device

sample data:

<DiagnosticTests>
  <DiagnosticTest id="12345678901234567890">
    <Patient>
      <Id>1234567890</Id>
      <Name>Jane Doe</Name>
      <Gender>Female</Gender>
      <Age>25</Age>
      <AgeUnit>years</AgeUnit>
      <PregnancyStatus>Not Pregnant</PregnancyStatus>
    </Patient>
    <SampleId>0987654321</SampleId>
    <Facility>Rural Clinic ABC</Facility>
    <TestDate>2015-05-18T12:34:56+05:00</TestDate>
    <Test type="RDT">
      <Code value="SD_MALPFPV_02_02" displayValue="SD Malaria P.f and P.v" />
      <Control qualitative="Valid" value="45.67" />
      <Assay code="HRPII" codingScheme="Fio" displayValue="P.f">
        <Result qualitative="Positive" value="23.45" />
      </Assay>
      <Assay code="pLDH" codingScheme="Fio" displayValue="P.v">
        <Result qualitative="Negative" value="0.0" />
      </Assay>
    </Test>
  </DiagnosticTest>
</DiagnosticTests>

use this:

<DiagnosticTests>
  <DiagnosticTest id="{general_id}">
    <Patient>
      <Id>{general_id}</Id>
      <Name>{system_user}</Name>
      <Gender>{gender_long}</Gender>
      <Age>{age}</Age>
      <AgeUnit>years</AgeUnit>
      <PregnancyStatus>Not Pregnant</PregnancyStatus>
    </Patient>
    <SampleId>{sample_id}</SampleId>
    <Facility>Rural Clinic {city}</Facility>
    <TestDate>{start_datetime}</TestDate>
    <Test type="RDT">
      <Code value="SD_MALPFPV_02_02" displayValue="SD Malaria P.f and P.v" />
      <Control qualitative="Valid" value="{decimal}" />
      <Assay code="HRPII" codingScheme="Fio" displayValue="P.f">
        <Result qualitative="Positive" value="{decimal}" />
      </Assay>
      <Assay code="pLDH" codingScheme="Fio" displayValue="P.v">
        <Result qualitative="Negative" value="0.0" />
      </Assay>
    </Test>
  </DiagnosticTest>
</DiagnosticTests>

The current supported replace tags are :

{result}
{sample_id}
{general_id}
{system_user}
{end_datetime}, {start_datetime}
{event_id}
{test_type}
{city}
{gender}
{gender_long}
{race}
{ethnicity}
{age}
{decimal}

To generate demo data automatically each day:

• enable demo data as shown above in step 1.

• for testing locally i normally change the cron time in japp/jobs/daily_demo_data.rb to run every few minutes.

view the results in http://localhost:3000/sidekiq/ , [note the added ‘cron’ tab in the header], and verify test_results count is increased in mysql.

• To test locally, start rails/sidekiq with foreman start

deploy onto EC2

• before ‘make’ change docker-dev.env to enable the demo data.

• to view the cron job to debug any issues it is useful to change routes.rb [remember to change back once you know it works]: mount Sidekiq::Web => '/sidekiq' if Rails.env == 'development'

• to mount Sidekiq::Web => '/sidekiq'

• change: app/jobs/daily_demo_data.rb to run each hour for testing

  Sidekiq::Cron::Job.create(name: 'Demo Data - daily 1am', cron: '* */1 * * *', klass: 'DailyDemoData') #run daily 1am

and how many test points per device you want

  def get_repeat_demo_per_device
      1000
  end

Note: to see logs from sidekiq:

docker-compose logs sidekiq