Webhooks (HTTP Callbacks)

Overview

Webhooks can be inserted between sections of a form to call an external service and perform workflow actions on the form based on the result of service call.

Webhooks will be created in the form HTML by creating a new <form> element at the point in the workflow where the service should be called.
When that “service section” becomes active (previous section approves or a future section returns) the service is called.
The service will receive all the data from all the form sections visible to the service section, plus some metadata on the form itself and the sections.
- The data sent will not have field-level visibility applied. There is no immediate plan to apply field-level visibility rules to data sent to services (issue #137) since in all the current use cases the form creator also controls the service. Please contact forms@northwestern.edu if you need to apply field-level visibility rules to data sent to web services defined in service sections.
Any data returned in the 'formcycle-data' structure by the web service will be stored in fields in the service section (e.g., for JavaScript or custom notification data substitution in future sections to act on).
- If data returned by the service does not match a field in the service section HTML the data will still be saved on the backend (e.g., for email alerts or future reporting) but it will not be populated when the service section is rendered.
Additionally the service should always tell the Online Forms engine what action to take on that section: approve, reject, save, return. Use the 'formcycle-action' field for this.
If the service cannot be reached, returns an error, or performs a save action the service will be retried until an approve, reject, or return action is issued. The frequency of retries will decrease over time.
If Online Forms is unable to connect to a web service defined in a service section for over an hour, or continues to receive responses it does not understand, then it will notify the group that owns the Form Template. If it continues failing then it will keep trying, and continue notifying the group once a day.
If Online forms receives responses from web services which are missing critical information, or contain bad information, then it will notify the group that owns the Form Template.

Defining a service section

To declare a service section do the following in a form tag.

The class attribute must contain 'formcycle-service-section'. In addition, like with any other form section, the class attribute must also contain 'form-section'. Any visibility rules should also be defined in the class attribute.
A 'formcycle-service-action' attribute must be included in the form tag. The URL of the web service must be HTTPS.
A 'formcycle-service-method' attribute must be included in the form tag. This attribute must be 'post' or 'POST'. We may support other HTTP methods in the future.
If HTTP basic auth is required to access the web service then a 'formcycle-service-user' and 'formcycle-service-password' attributes must be included in the form tag.

Any service section related attributes will be scrubbed from the HTML sent to a user's browser even if the service section is set to be visible to users. Users will not be able to discover your web service URLs or HTTP basic auth credentials this way.

An example service section declaration can be seen below. In this particular case the service section is visible from another section, named "Student". The service section could include some visible data fields, which could receive data from the service section call, possibly informing the submitter about how the call to the service went.

<form id="Eligibility_Check" class="form-section visiblefrom-Student formcycle-service-section" formcycle-service-action="https://webapps6-test.weinberg.northwestern.edu/forms-testservices/approve.php" formcycle-service-method="post" formcycle-service-user="test-user" formcycle-service-password="test-pass">
	<!-- Include some fields here if desired -->
</form>

Data sent by service sections

The JSON data payload POSTed to a web service by a service section includes information about all of the sections related to the form submission, limited to ones visible to the service section. Also included is information about the related section templates, form version, and form template. Any users or groups assigned any of the sections are also included. Below is an example of the data submitted with a service call from Online Forms. Note that the section names of "Part1-of-form" and "Part2" just happen to be the section names used in the example form, these can be anything you define in your form.

{
   "FormVersion":{
      "id":"266",
      "active":true,
      "form_template_id":"162"
   },
   "FormTemplate":{
      "id":"162",
      "name":"Service Section Testing Form",
      "group_id":"1",
      "Group":{
         "id":"1",
         "name":"Weinberg IT",
         "displayname":"Weinberg IT",
         "organization_id":"1",
         "Organization":{
            "id":"1",
            "name":"Weinberg College of Arts and Sciences"
         }
      }
   },
   "Sections":{
      "Part1-of-form":{
         "SectionInstance":{
            "id":"1519",
            "created":"2015-03-26 13:41:38",
            "modified":"2015-03-26 13:41:38",
            "data":{
               "Part1-of-form":{
                  "Date":"3/26/2015",
                  "Student_ID":"123",
                  "First_Name":"Samuel",
                  "Middle_Initial":"X",
                  "Last_Name":"Tudent"
               }
            },
            "section_template_id":"682",
            "user_id":"9",
            "last_saved_by_user_id":"9",
            "parent_section_instance_id":null,
            "approved":true,
            "rejected":false,
            "returned":false,
            "archived":false,
            "optional_not_activated":false,
            "ready":false,
            "group_id":"0",
            "User":{
               "id":"9",
               "firstname":"Samuel",
               "lastname":"Tudent",
               "displayname":"Samuel Tudent",
               "netid":"student"
            },
            "LastSavedByUser":{
               "id":"9",
               "firstname":"Samuel",
               "lastname":"Tudent",
               "displayname":"Samuel Tudent",
               "netid":"student"
            },
            "Group":{
               "id":null,
               "name":null,
               "displayname":null,
               "organization_id":null
            }
         },
         "SectionTemplate":{
            "id":"682",
            "name":"Part1-of-form",
            "form_version_id":"266",
            "order":"1",
            "from_email_address":null,
            "from_email_name":null
         },
         "Attachment":[
         ]
      },
      "Part2":{
         "SectionInstance":{
            "id":"1520",
            "created":"2015-03-26 13:41:38",
            "modified":"2015-03-26 13:41:38",
            "data":[
            ],
            "section_template_id":"683",
            "user_id":"0",
            "last_saved_by_user_id":"0",
            "parent_section_instance_id":"1519",
            "approved":false,
            "rejected":false,
            "returned":false,
            "archived":false,
            "optional_not_activated":false,
            "ready":true,
            "group_id":"0",
            "User":{
               "id":null,
               "firstname":null,
               "lastname":null,
               "displayname":null,
               "netid":null
            },
            "LastSavedByUser":{
               "id":null,
               "firstname":null,
               "lastname":null,
               "displayname":null,
               "netid":null
            },
            "Group":{
               "id":null,
               "name":null,
               "displayname":null,
               "organization_id":null
            }
         },
         "SectionTemplate":{
            "id":"683",
            "name":"Part2",
            "form_version_id":"266",
            "order":"2",
            "from_email_address":null,
            "from_email_name":null
         },
         "Attachment":[
         ]
      },
      "Part3":{
         "SectionInstance":{
            "id":"1521",
            "created":"2015-03-26 13:41:38",
            "modified":"2015-03-26 13:41:38",
            "data":[
            ],
            "section_template_id":"684",
            "user_id":"0",
            "last_saved_by_user_id":"0",
            "parent_section_instance_id":"1519",
            "approved":false,
            "rejected":false,
            "returned":false,
            "archived":false,
            "optional_not_activated":false,
            "ready":false,
            "group_id":"0",
            "User":{
               "id":null,
               "firstname":null,
               "lastname":null,
               "displayname":null,
               "netid":null
            },
            "LastSavedByUser":{
               "id":null,
               "firstname":null,
               "lastname":null,
               "displayname":null,
               "netid":null
            },
            "Group":{
               "id":null,
               "name":null,
               "displayname":null,
               "organization_id":null
            }
         },
         "SectionTemplate":{
            "id":"684",
            "name":"Part3",
            "form_version_id":"266",
            "order":"3",
            "from_email_address":null,
            "from_email_name":null
         },
         "Attachment":[
         ]
      }
   }
}

Return data format for web services called by service sections

The web services called by service sections must respond in certain ways to allow Online Forms to understand what should occur with the returned data and the workflow of the form. There may also be additional data fields that a creator of a web service called by service sections should consider returning. Below is a list of things a web service must and should do:

It must respond with JSON data.
It must respond with an appropriate HTTP status code.
It must also include that HTTP status code in the JSON data using the 'status' key.
It should include a recommended workflow action in the JSON data using the 'formcycle-action' key. Valid options are 'approve', 'reject', 'return', 'save'.
If no workflow action is recommended and the HTTP status code is 200, then Online Forms assumes the action to be 'approve'. If any other HTTP status code is returned, then a 'save' action is performed and the system will make the request again after a waiting period.
If a return action is recommended then Online Forms must be told what section instance id to return the workflow to using the 'formcycle-return-section-instance-id' key, and a return reason should be provided in the JSON data using the 'formcycle-return-reason' key. The return reason will be visible to the person(s) assigned to the section the form is returned to, and possibly to others if a custom notification template includes the return reason. By providing a helpful return reason the assignee(s) are more likely to be able to fix their submission to allow it to be approved in the future. The section instance id must be a valid section instance id that is part of the form submission, and must be an id belonging to a section instance earlier in the workflow.
If a reject action is recommended then a reject reason should be provided in the JSON data using the 'formcycle-reject-reason' key. The reject reason will be visible to the person who originally submitted the form, and possibly to others if a custom notification template includes the reject reason.
If the web service wants any data saved in the service section data area, then data fields and values should be provided in the JSON data using the 'formcycle-data' key. Any data provided in formcycle-data using a key that matches a field name in the form's service section can be visible to users of other sections, if the service section's visibility rules allow it. Any data provided in formcycle-data using a key that doesn't match a field name in the form's service section can only be accessed using a notification template's data merging {{section_data.SECTIONAME.FIELDNAME}} feature, or possibly using future reporting functionality.

Return data format for a web service called by Online Forms.

 {
   "status":HTTP_STATUS_CODE,
   "formcycle-action":"ACTION",
   "formcycle-reject-reason":"REASON", (optional, only for formcycle-action = reject)
   "formcycle-return-reason":"REASON", (optional, only for formcycle-action = return)
   "formcycle-return-section-instance-id":"SECTION INSTANCE ID", (required, only for formcycle-action = return)
   "formcycle-data":{				   (optional, only if web service wants data saved in section)
      "FIELDNAME":"DATA", 
      "FIELDNAME":"DATA",          
      "FIELDNAME":"DATA"
   }
}

Test web services available to Online Form creators

To make the testing of webhooks / services sections in Online Forms easier we have made several test web services available. These will each return a specific formcycle-action, and formcycle-data. These services are available from anywhere on the NU network (129.105.0.0/16, 165.124.0.0/16, 10.0.0.0/8). Do not POST any sensitive data to these test services, as anything POSTed to them will be recorded in a public log file for debug purposes.

Service	Description	Service URL	Service Log file	Notes

Service	Description	Service URL	Service Log file
Approve	This service always approves any request made to it	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/approve.php	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/logs/approve.log
Save	This service always suggests a save for any request made to it	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/save.php	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/logs/save.log
Return	This service always suggests a return to the first section of a form for any request made to it	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/return.php	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/logs/return.log
Reject	This service always suggests a rejection for any request made to it	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/reject.php	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/logs/reject.log
Bad response	This service always returns a bad response, causing Online Forms to queue the service section action and try again.	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/bad-response.php	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/logs/bad-response.log
503	This service always returns a HTTP 503, service unavailable, causing Online Forms to queue the service section action and try again	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/503.php	https://webapps6-test.weinberg.northwestern.edu/forms-testservices/logs/503.log

All of the test web services will return a HTTP 405 Unsupported Method if you use any HTTP method other than POST.

There is nothing special about the 'usermsg' field in formcycle-data in these services, the field could be named anything in your own service. However, if you call the test services from your own test form and want to see the returned message, you will have to have a field named usermsg in your service section.

Example returned data from a POST to approve.php test service:

Example returned data from a POST to save.php test service:

Example returned data from a POST to return.php test service. Note that the formcycle-return-section-instance-id field value will be the section instance tied to the first section template (field 'order' = 1) in the form. If return.php is unable to figure out which section instance is the first one it will instead issue a reject action, a reject-reason and usermsg explaining why.

Example returned data from a POST to return.php test service, where it was unable to determine what section to return the form to. In this case it issues a reject instead.

Example returned data from a POST to reject.php test service:

Example returned data from a POST to 503.php:

Example returned data from a POST to bad-reponse.php: