Comparing xTesting v. Robot Framework

# Comparing xTesting v. Robot Framework(Robot FW) ###### tags: `osc` `int` :::info **Outline:** [TOC] ::: :::info #### Purpose: 1. Goal is to make sure maintainability of **it/test** repo 2. Find out which format should be maintained between xTesting and Robot FW ::: :::warning #### Issue: 1. Robot FW test file can only be run using the Robot FW cli. 2. Robot FW test file is incompatible with xTesting cli. 3. xTesting test file is incompatible with Robot FW cli. ::: #### **Table comparison of xTesting v Robot** | | Xtesting | Robot | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | Repo | https://github.com/opnfv/functest-xtesting | https://robotframework.org/ | | Last commit | Oct 25, 2024 | Dec 11, 2024 | | Release | master | [release tag: Robot Framework 7.1.1](https://github.com/robotframework/robotframework/releases/tag/v7.1.1) | | Documentation | - https://lf-anuket.atlassian.net/wiki/spaces/HOME/pages/21856527/Xtesting - https://docs.onap.org/projects/onap-integration/en/latest/integration-tooling.html | https://robotframework.org/ | | Description | - xtesting is a CI/CD framework, neither a test nor a test framework -Xtesting is a python library harmonizing the way to setup, run, teardown, manage the artifacts, manage the reporting of the tests | automation framework for test automation | | Project adoption | - AIMLFW(have not tested) - RICPLT (basic health check) | - RICPLT(outdated) - O2 IMS | | Syntax/Format | yaml (ansible) | robot syntax | | Execution | `ansible-playbook test_file.robot` | `robot test_file.robot` | ## Compatibility test - Run same file with different cli ### :page_facing_up: File test: [7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot](#7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot) :::info path: [test/test_scripts/O2IMS_Compliance_Test/o2ims_compliance/7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot](#7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot) ::: 1. Run with `robot` cli :::warning cmd: ```shell= robot 7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot ``` output: ``` 18 tests, 0 passed, 18 failed ============================================================================== Output: /home/ubuntu/test/test_scripts/O2IMS_Compliance_Test/o2ims_compliance/output.xml Log: /home/ubuntu/test/test_scripts/O2IMS_Compliance_Test/o2ims_compliance/log.html Report: /home/ubuntu/test/test_scripts/O2IMS_Compliance_Test/o2ims_compliance/report.html ``` ::: 2. Run with `ansible-playbook` cli :::warning cmd: ```shell= ansible-playbook 7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot ``` output: ``` [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' ERROR! We were unable to read either as JSON nor YAML, these are the errors we got from each: JSON: Expecting value: line 1 column 1 (char 0) Syntax Error while loading YAML. did not find expected alphabetic or numeric character The error appears to be in '/home/ubuntu/test/test_scripts/O2IMS_Compliance_Test/o2ims_compliance/7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot': line 1, column 2, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: *** Settings *** ^ here ``` ::: ### :page_facing_up:File-test: [health-check.robot](#health-checkrobot) :::info Path: [/test/XTesting/XTesting-demo/health-check.robot](#health-checkrobot) ::: 1. Run with `robot` cli :::warning cmd: ``` robot healthcheck.robot ``` output: ``` [ ERROR ] Suite 'Healthcheck' contains no tests or tasks. Try --help for usage information. ``` ::: 2. Run with `ansible-playbook` cli :::warning cmd: ```shell= ansible-playbook healthcheck.robot ``` output: ``` [WARNING]: No inventory was parsed, only implicit localhost is available [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' PLAY [Shell module example] ************************************************************************************************** TASK [Gathering Facts] ******************************************************************************************************* ok: [127.0.0.1] TASK [Check system information] ********************************************************************************************** fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": "curl -v http://TARGET-IP:32080/appmgr/ric/v1/health/ready 2>&1", "delta": "0:00:00.023701", "end": "2024-12-13 04:25:12.100841", "msg": "non-zero return code", "rc": 6, "start": "2024-12-13 04:25:12.077140", "stderr": "", "stderr_lines": [], "stdout": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Could not resolve host: TARGET-IP\n* Closing connection 0\ncurl: (6) Could not resolve host: TARGET-IP", "stdout_lines": [" % Total % Received % Xferd Average Speed Time Time Time Current", " Dload Upload Total Spent Left Speed", "", " 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Could not resolve host: TARGET-IP", "* Closing connection 0", "curl: (6) Could not resolve host: TARGET-IP"]} PLAY RECAP ******************************************************************************************************************* 127.0.0.1 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 ``` ::: ## Syntax/Format differences in file ### 7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot :::info Link to gerrit: [click here](https://gerrit.o-ran-sc.org/r/gitweb?p=it/test.git;a=blob;f=test_scripts/O2IMS_Compliance_Test/o2ims_compliance/7.3.2_SMO_succeeds_to_query_inventory_with_O2ims_APIs_and_correct_token.robot;h=7f2336b084b8bcbc9e2ac53aa00eebfaeeae84ab;hb=HEAD) ::: ``` *** Settings *** Documentation Keywords for interacting with the A1 interface, including policy creation, instantiaton, and deletion Library RequestsLibrary Resource /robot/resources/global_properties.robot *** Variables *** ${A1MEDIATOR_BASE_PATH} /a1-p/policytypes ${A1MEDIATOR_ENDPOINT} ${GLOBAL_A1MEDIATOR_SERVER_PROTOCOL}://${GLOBAL_INJECTED_A1MEDIATOR_IP_ADDR}:${GLOBAL_A1MEDIATOR_SERVER_PORT} *** Keywords *** Create A1 Policy Type [Documentation] Create a new policy via the A1 Mediator. [Arguments] ${type} ${name} ${description} ${properties} ${required}=@{EMPTY} ${typeID} = Convert To Integer ${type} Should Be True ${type} > 0 Policy type must be an integer > 0 ${createSchema} = Create Dictionary ... $schema=http://json-schema.org/draft-07/schema# ... type=object ... properties=${properties} ... required=@{required} ${createBody} = Create Dictionary ... name=${name} ... policy_type_id=${typeID} ... description=${description} ... create_schema=${createSchema} ${createJSON} = Evaluate json.dumps(&{createBody}) json,uuid ${resp} = Run A1Mediator PUT Request /${type} body=${createJSON} [Return] ${resp} Instantiate A1 Policy [Documentation] Create a new instance of an A1 policy [Arguments] ${type} ${instance} ${properties}=${EMPTY} ${typeID} = Convert To Integer ${type} Should Be True ${type} > 0 Policy type must be an integer > 0 ${instanceJSON} = Evaluate json.dumps(&{properties}) json,uuid ${resp} = Run A1Mediator PUT Request /${type}/policies/${instance} body=${instanceJSON} [Return] ${resp} ``` ### health-check.robot :::info Link to gerrit: [click here](https://gerrit.o-ran-sc.org/r/gitweb?p=it/test.git;a=blob;f=XTesting/XTesting-demo/healthcheck.robot;h=7014c0a725cf514f56585b123f64a6d01a20d978;hb=HEAD) ::: ``` --- - name: Shell module example hosts: 127.0.0.1 tasks: - name: Check system information shell: "curl -v http://TARGET-IP:32080/appmgr/ric/v1/health/ready 2>&1" register: os_info - debug: msg: "{{os_info.stdout_lines}}" ``` ## run_tests [This is part of Openstack project to run tests](https://docs.openstack.org/horizon/ocata/ref/run_tests.html). xTesting will install run_tests which contain: ``` #!/usr/bin/python3 # -*- coding: utf-8 -*- import re import sys from xtesting.ci.run_tests import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(main()) ``` :::warning run_tests already replace by [tox](https://tox.readthedocs.io/en/latest/) :::