SparkLabs

We believe that passion drives great ideas. Our teams are very passionate about the work they do, which results in a lot of innovative ideas. SparkLabs is where we explore those ideas, and develop them into tools and products that bring value to our services, and ultimately our customers.

We have developed a structure of Innovation Centers that are carefully designed to support and improve our services. The Centers serve as a knowledge base, and analyze our experiences and lessons learned to identify opportunities for improvement. Innovation Centers feed into our work in SparkLabs, and have produced process improvement concepts like DevTOps, which you can read more about below. Click on one of our Centers to learn more about what they do.

MockSpider logo platform-agnostic data solution

Data Pipelines are Expensive to Maintain and Rigid by Nature.

We found that all analytical systems deal with the same burdens caused by data ingestion. In addition, these systems are never able to consistently communicate the data transformation rules applied to the data in terms that the business user community can understand. This inability to communicate data transformation rules results in an even greater need to rely on technical staff to address these business-driven inquiries.

So we created our eleVate solution.

eleVate (Extract, Load, Enhance, and Virtually Transform) is a platform-agnostic solution that leverages dynamic SQL to profile, improve, and prepare incoming data for future analytics. eleVate addresses common problems upfront while allowing systems to ingest self-described data without custom code. Time-versioned views are applied to implement any remaining complex business rules. In addition to nearly eliminating source dependencies, eleVate also allows users to re-run their analytics against the system as it was in the past.

Features

  • A platform-agnostic solution built using ANSI-Standard SQL.
  • Reacts to incoming file layouts guided by user input to rapidly deliver improvements.
  • Transformed data resides alongside raw data.
  • Modular dynamic improvement transformations ensure consistent transformations across your whole system.
  • Complex business logic allows for historic what/if analytics.
  • Provides daily configurable data profiling and trend analysis across all incoming data sets.
  • Addresses issues including: data ingestion, referential integrity, duplicate data, bad data typing, non-contiguous overlapping dates, lists of values, field and table aliases, and more.

MockSpider logo
Demonstration

Step 1: Log in to eleVate and upload a file

  1. Users click login (Login values are pre-populated with (spark/spark123)
  2. Login send you to: https://snowspark-dev.sparksoftcorp.net/index
  3. Where you choose upload → Choose File → Open → Submit and select and upload either a csv,json or xml file. You will see a message saying the request has been submitted. Click on the Dashboard to see if your file load has completed.

Step 2: Click here to load Snowflake and explore the results

Login with user: READ_TEST_DEMO & Password: Spark123
Run the following SQL in the Snowflake interface to explore the results:

  • Replace FilenameWithoutExtension with the name of your uploaded file without the extension.
  • Highlight each query starting with “SELECT” ending at “;” and click the Run button to execute following queries to show the results:
-- FIRST SPECIFY THE WAREHOUSE:

USE WAREHOUSE DEMO_WH;

-- To see the results of the process:

-- For example:

-- SELECT *

-- FROM DEMO_DB.TEST_DEMO.PROVIDER;

 

SELECT *

FROM DEMO_DB.TEST_DEMO.FilenameWithoutExtension

;

-- To see the results of the column level data profile.:

-- For example:

-- SELECT *

--FROM DEMO_DB.TEST_DEMO.V_IVT_METADATA_TBL_CLMN_RPT

--WHERE TGT_TBL_NAME = 'PROVIDER';

 

SELECT *

FROM DEMO_DB.TEST_DEMO.V_IVT_METADATA_TBL_CLMN_RPT

WHERE TGT_TBL_NAME = 'FilenameWithoutExtension'

;

-- To see the details related to extraction and load.:

-- For example:

-- SELECT *

--FROM DEMO_DB.TEST_DEMO.V_IVT_METADATA_TBL_JOB_RPT

--WHERE TGT_TBL_NAME = 'PROVIDER';

 

SELECT *

FROM DEMO_DB.TEST_DEMO.V_IVT_METADATA_TBL_JOB_RPT

WHERE TGT_TBL_NAME = 'FilenameWithoutExtension'

;

Note 1: If the file name started with a numeric or special character, the process will adjust the target table name by applying an ‘SRC_’ prefix to it.

Note 2: This is a shared demo environment, which means multiple people might have updated files.

Note 3: Files are removed daily.

MockSpider logo virtual integration tool

Testing is critical.

It is also complex, time consuming, and expensive. Loading data is slow, and you can't test during this time. Further delays are caused by waiting for all of the components to be completed before testing. This lost time adds up quickly, and is costly. We knew that we could develop a way to bring efficiency and flexibility to this cumbersome process.

So we created MockSpider.

MockSpider provides a virtual environment for your data, and creates a "sandbox" for developers to test in. Data can be batch uploaded, and is available immediately to begin testing. Your data is infinitely reusable, and available to multiple systems.

MockSpider can also simulate incomplete components, allowing you to test your completed components earlier. Developing and testing concurrently makes finding and fixing bugs easier, and significantly shortens the lifecycle of your project.

Now, testing is fast, flexible, and affordable.

Features

  • Provided as Software as a Service (SaaS), it is easily scalable.
  • You don't have to develop and manage web service sandboxes, MockSpider takes care of it all for you.
  • You can choose to use our secure cloud hosting, or install it in your data center.
  • It mocks services for SOAP and REST.
  • Virtualized data is available for testing immediately, infinitely reusable, and accessible to multiple testing systems.
  • Incomplete components of your application are simulated, so completed components can be tested earlier.
  • You can mock services quickly and easily using Smart Templates.
  • Enterprise level role-based access controls access to information.
reusable testing framework

Testing is labor intensive.

We have run extensive testing, both manual and automated, on large and incredibly complex systems. Through this we have identified pain points in the testing process, and gained valuable understanding of the unique needs of these processes. We knew there had to be a way to streamline testing.

So we created ScriptHive.

ScriptHive is a hybrid automation testing framework, supporting both Test Driven and Business Driven Development. More than that, it is reusable and customizable. You set ScriptHive up to run your test scripts for a project, and at the end you can re-tailor the framework for use with your next project, eliminating the need to build a new framework each time.

ScriptHive has been built to run grid-based parallel execution, in a cloud environment. This allows you to run multiple tests across browsers,concurrently, and without the need for physical machines. We also added features for scheduling tests, monitoring, recovery, and automatically generated and shared reporting.

Features

  • Hybrid framework for both Test and Business Driven Development.
  • Reusable, flexible, and customizable to each project.
  • Cloud based execution eliminates the dependence on physical machines for testing.
  • Automatic script triggering allows you to schedule testing, without manual supervision.
  • Automatically generates and shares test results.
  • Built to run parallel execution, across browsers.

accessibility validation plug-in

508 compliance is important.

508 compliance is not a new requirement, but standards remain poorly understood by many, and testing is often performed as an afterthought.

We decided to simplify 508 testing, and make it easier. The solution needed to be fast, flexible, and able to stand alone or integrate into a current system.

So we created 508bee.

508bee is a browser plug-in and API that tests websites and documents for 508b and WCAG AA standards compliance.

Define what you want to test (a single page, the whole site, a user path, etc.), and 508bee will automatically check the code against its database of standards rules. It returns reports that are easy to understand, and built to help developers fix the code quickly.

508bee can be integrated into your development and testing environments, and modified with custom scripts to tailor its performance. The API easily integrates with Selenium frameworks, making 508bee an integral part of automation testing. This encourages testing early and often, and brings awareness of 508 compliance issues into the whole lifecycle.

Don't just test, develop for compliance

Features

  • Plug-in format makes it easy to test right from the browser.
  • Tests static and dynamic pages, in both standard and responsive sites.
  • Has a Developer mode that highlights web elements, and allows the developer to fix the issues in debug mode.
  • Reports are designed for developers, to help fix the code faster.
  • Tests documents, such as PDFs, hosted on your site or uploaded through the web portal.
  • Can be integrated with your Content Management System, and customized with scripts.
  • Tests both 508b and WCAG AA rules.


test-centric CI/CD pipeline

As part of our push for innovation, we have developed an update to the DevOps concept. We adopt an Agile approach and use testing to bridge the gap between development and operations. DevTOps embeds testing into the lifecycle, testing the system during development. This brings a greater ability to handle change, identifies defects early, and accelerates testing overall. We’ve drawn from years of field experience, analyzing strategies that positively impact quality and success, and building in a balance of flexibility and consistency.

For more information about these products and the latest Sparksoft innovations,
call 410-424-7700 or info@sparksoftcorp.com

Sparksoft's Snowflake Data Marketplace Disclaimer