[Epic] JSON Form: Generate forms from JSON Object/Queries/APIs

[areyabhishek]

Objective

As a user, I should be able to generate a form with input fields by using fields of a DB table schema that automatically generates the queries, bindings and child widgets. As a user, I should be able to configure changes/validations easily here.

Features

• It should automatically identify the datatypes and associated widgets for the form
• It should be able to add basic validations that can be enhanced by the user
• It should be able to automatically connect the submit buttons to the relevant queries (PUT/POST/CREATE/UPDATE).

Success Metrics

Goal	Metric
Simplify user experience	One-click to generate a form from a single DB schema and be able to map it to a PUT/POST request which updates/saves data
Decrease the time needed to build a form	Time that user spends creating a form

Assumptions

• Forms are the core building block of a web application
• It is currently difficult/time-taking to set up a working form comprising of multiple child widgets and bind them all the data query and finally be able to add them to a post request
• One of the most common operations in any DB is the insert/add/create entry operation. It would go a long way to save the user time by generating the scaffolding, if not the complete widget for users

Requirements

Requirement	User Story	Importance	Notes
Connecting to Query/API	As a user, I should be provided with all the option to connect to different queries and APIs. Furthermore, I should be able to select the fields I need inside the form and I should be able to change the datatype to others from what was automatically connected	High	If this is not going to be based on the response object and depend on the DB schema, this will be removed from the scope
Create a form from a JSON object	As a user, I should be provided with all the option to create a form from a JSON object apart from the Queries/APIs. All the other functionalities mentioned above will be extended here as well	High	-
Identify datatypes and associated widgets	As a user, after connecting a DB query to the form, the platform needs to identify the widget types based on the data types. For ex: Array maps to Select widget. All the widgets need to remain configurable	High	related to the first requirement
Basic input validation	As a user, I should be able to see basic input validations for the form I have created from an internal DB query and also be able to modify any of these easily	High	related to the first requirement
Connect submit action to queries that update related DB schema	As a user, I should be able to easily configure the submit request which finally updates or saves my data. I should be shown all fields preloaded in my query and appropriate bindings against them	High	This might not be involved in this scope
Auto naming with schema / object keys	All fields & label in the form should be named based on the schema key they are mapped to	High	-
Reopen form config	Once import all fields from an object/datasource I should be able to reopen the config page view to alter any initial configurations. This will reset and changes made to the form elements individually.	High	The default connect data will be used for this, another CTA might be required to update the form data through the same workflow

Discussion 21-Sept-2021

• We will get rid of the modal which shows up in when setting up the form fields, Instead we show the fields under the field configuration directly
• Fields on the form will be deep linked to the property pane field configuration, i.e. the respective property pane field setting should open when the field on the form is clicked
• The form fields / view will reflect changes as the form data is updates
• We need to keep in mind a future scope while developing the feature. Nesting of form fields / handling nested form data objects needs design **_pending_**

Discussion 27-Sept-2021

We decided that the following will be the type of widget taking into consideration, Additionally the component used in this widget will have there own set of properties (like the table widget) which should be covered by automated tests to avoid any regressions in the future. We also discussed the form properties which will need design review

Types of fields:

• Text
• Number
• Phone
• Email
• Switch (boolean)
• Checkbox (boolean)
• RadioGroup (exclusive or)
• Select (one out of many)
• Multi-Select (many out of many)
• Files
• Date
• Time
  ​
  Complex fields:
• Grouped fields
  ​
  Additional (optional) fields:
• Rate (Range)
• Map (search)
• JSON object
• Audio

---

We figured Properties of fields:

Common:

• Type
• Default
• Label
• Disabled
• Required
• Tooltip
• Visible
• helperText
  ​
  Not common:
• Validation Fn
• Error
• Possible values (options)
• Placeholder
• onChange
  ​

---

**_needs design_**
Main properties of the form for it to work:
Form: data
Form: submit (action)
Form: source data
Form: Edit the field types and properties
​​
Features in the form:
Form -> input type -> possible properties
Form -> has state of the fields
Form -> onSubmit
Form -> onCancel
Form -> can hide cancel button
Form -> can disable submit button
Form -> can reset itself*
Form -> checks for validations -> knows which fields have errors
Form -> tab navigation works
Form -> can scroll to selected fields
Form -> button location (sticky | bottom)
Form -> field styling (borders, colors, label alignment, error colors, helper text, etc)
Form -> spacing between fields
Form -> all fields have full width

---

Library to consider: react-hook-forms

---

Discussion 19-Oct-2021

Files component is being considered out of scope right now as the implementation of the widget component is not well constructed to be used in this widget and would need a lot of refactoring.
This can be considered in the next iteration.

Out of Scope

• Dynamic width / auto layout of form elements
• Alignment templates/Layouts - As a dev, I should be able to choose the layout for aligning my input fields in the form. Templates can be 2 Column, Single column, Label & input side by side. Or, Drag to re-order elements side by side or one below the other

Developer Handoff Document in Figma

Link: https://www.figma.com/file/pZuLD9mWXtk45TOjAeh32b/Form-Widget-Thu-26-Aug?node-id=2%3A2

Questions

Question	Answer	Date Answered
How to identify the submit request needs to be Create/Update?	It will by default be POST the user can change to there choice
Can we add necessary formatting function ex: JSON.stringify() automatically to remove any invalid api attributes

[naveennazimudeen]

It will be great if this can be combined with #2576 to show related information below. Tables have relationship, sometimes expressed in database as foreign keys, sometimes relationships exist but is deliberately not defined as foreign key relationships in DB. For example Invoices have a 1:n relationship with line-items, line-items might have 1:1 relationship to product. Similarly for Users and Logins, for Companies and Contacts, Subscription and Payments and so on.

So when defining an invoice page (View Page), it should be possible to see Invoice Line items as table below, where you can possibly filter, search, create (line by line, or go to new page), and edit (line by line, or go to individual edit page). The product column can possibly be linked to the product view page. While creating a line-item, it should be possible to "search as you type" for product (1:1 relation) with an option to create a product (in a modal window) if one does not exist .

Relationships can be captured independent of DB Foreign keys in appsmith, may be when a query is defined for view page, users can define more queries for relationship records, with an option to define 1:1 relationship columns such as productId from the invoice example.

[somangshu]

#1724 is a related issue. Can one of them be closed?

[Nikhil-Nandagopal]

Why is

The form can be generated from the JSON object

out of scope? That should be the core of it since to generate the form from the DB schema, we need a JSON object right?

[somangshu]

Just for the purpose of the design phases, You are right though! I will add it to the list of requirements!

[somangshu]

Create granular tasks after design is completed.

[rishabhsaxena]

Just to pitch in, I was wondering if we have any plans to do wizard forms, basically a form with multiple steps

[Nikhil-Nandagopal]

@rishabhsaxena great point. Thought might need to be a separate widget. Could you raise an issue for a wizard?

[rishabhsaxena]

@Nikhil-Nandagopal Yeah just created an issue for this #6351

[somangshu]

After a brief discussion, Here is what we understand:

• Showing all fields as tile inside the property pane (like table widget) can be a viable option, However this is does not let the user see and change all config together and might be tedious to update.
• Showing all options in the modal can be better but will be an inconsistent experience
• There are two ways to let the user select layout
  • Through a separate widget (column / layout widget)
  • Giving a few custom options while setting up the form (1 column or 2 column)
• Technically we feel that once the fields are organised and finalised we can do the following:
  • Let the user drag and drop anything they'd like without any constrain, but the initial settings are no longer available. In case they wish to start again, there changes are going to be lost
  • The user is not allowed to D&D anything directly and only allow them to do this via config panel (probably have a separate widget in this case - [Feature] JSON Form Widget #1724)

@momcilo-appsmith @riodeuno please add anything i might be missing here.

cc: @areyabhishek @Nikhil-Nandagopal

Reference form builder

[tomjose92]

Hey team, any timeline on this. This would be a great feature to have

[somangshu]

@tomjose92 we are in the process are getting designs done for this feature, We will be taking this up for implementation in the month of sept.

[ashit-rath]

Action Items:

• Solve for the best way to sanitize/represent JSON keys that cannot be referenced with a dot (.) notation (eg "first name")

[dilippitchika]

For the clear functionality -
I was finding any existing bugs linked to form widget where users wanted to keep/remove defaults on click of reset. Found one in support of keeping defaults #4230. I believe removing defaults can be asked by devs at somepoint, but let's wait to see any evidence of this ask and then decide if we want it.

When we decide to add the functionality, this is what the change may be

1. Reset a widget will have another option to set to empty vs default. This can be used by all buttons and not just form buttons.
2. There should be one button element which can be used in a regular case and form case. Today there’s no binding on property pane for form reset button. Even if you turn off reset on success, it still goes ahead and restores defaults after you click. This binding today is implicit i think. We are planning to solve this [Feature]-[800]:Deprecate form button widget #3613

[somangshu]

@dilippitchika #3613 is about merging form and normal button and have one for them, which can be used in both places, And henceforth reducing the chance of being error prone.

I still think that reset should go back to the default set, I am also pointing out to the crud page functionality we have, Where reset goes to default, This is very much expected. Even in this case.

[dilippitchika]

Second that @somangshu, just pointing out what we may need to do if we end up implementing it someday. Not doing the clear functionality for now.

[ashit-rath]

Action Items:
Required field and accordion updated designs @vasanth-appsmith
Sanitize source data keys similar to Table widget V2 @ashit-rath
Test Plan @vivekverma2312

[mmkal]

Has https://react-jsonschema-form.readthedocs.io been considered for this? It would cover the majority of use cases, but I'd imagine would take hours/days to implement rather than weeks/months/years. You just install the library! It might be worth a separate issue because it seems like there's a little more in scope here. Rjsf wouldn't know about database schemas out of the box, of course, but there would still be huge value in it. Coming from retool, we have a subsection of our api which we map with zero effort to useable UI forms for ops people. I'm new to appsmith, but familiar with react and typescript - is there a way we could use custom react components right now? If so we could probably get it working then contribute a PR.

[somangshu]

Hey @mmkal thanks a lot for showing interest on this feature. Yes indeed we have considered this package while building our own JSON Editor. Keeping the usefulness of this feature for the community in mind, we felt we needed customised experience, and hence we went about building stuff by ourself, We are near completion for v1 and would love to have your feedback, Please checkout this deploy preview to try out the feature, You can use any dummy credentials to login.

[Nikhil-Nandagopal]

@mmkal just to add to what somangshu means, react-jsonschema-form requires our users to learn a whole new schema language which we felt was too cumbersome. With our current implementation, you can just connect any object and the form will automatically render. You can then customise the properties of every field visually without needing to learn a schema

[mmkal]

I see, that makes sense, thank you for the response. In the case I'm hoping to successfully port from retool, we have json schema objects from the api already, and we don't have json data objects to work with. It sounds like this is a different use case though, so I think I'll do some more exploring and create a separate issue (or PR).

[somangshu]

Actionable:

• Documentation required @dilippitchika @Pranay105
• Can start review @riodeuno @ashit-rath @sbalaji1192
• can start QA @vivekverma2312 @yatinappsmith
• Some new widgets have gone into release, so need to port them @ashit-rath (feat: Input widget v2, currency widget & phone input widget #9582)
• test are written and done with
• Some issues pointed out by @dilippitchika needs to be dealt with - @ashit-rath
• Add sentry events to track the usage of JSON form @ashit-rath
• Should we have the onFocus / onBlur events for all the widgets? @dilippitchika

---

• Understand scope v2 @dilippitchika @somangshu
• after release update the CRUD templates to use this widget @abhvsn

[ashit-rath]

Actionables:

• Performance profiling @ashit-rath
• QA @vivekverma2312
• Porting of Select & multi-select V2 @ashit-rath @Tooluloope ([Enhancement]: Select and Multi-select V2 to handle server side filtering #10297)
• Code Review @sbalaji1192

[ashit-rath]

Performance issue update:
We had a discussion around the performance of the JSON Form widget when a large source data is used. The render time seems to come around 60 secs if we try to have ~150 fields (mix of arrays and objects) and out of ~60sec evaluations are taking around ~50secs; this is only for the first render.

Next steps:

1. Triage the cause of evaluations taking ~50secs for evaluating the schema.
2. We would be having a limit to how many fields the users are allowed to have in a form as having 200+ fields may lead the app to be in an unrecoverable state.

[ashit-rath]

Actionables:

• Check with Ajinkya if they can allocate someone to debug on evaluations performance issue. @ashit-rath
• Form limits - We will keep no. of items at 50 and use design 4 to communicate to the user - pending design change. @dilippitchika
• Next steps in JSON form - We need to understand the effort in each item in the scope, @dilippitchika to kick off this discussion offline.
• QA @vivekverma2312

[ashit-rath]

Actionables:

1. Setup an app-building exercise using JSON form and collect feedback. @ashit-rath
2. Add default source data @ashit-rath
3. Modify the limit message for the Appsmith app viewers @ashit-rath
4. Discuss ways to cache/retain fields to reduce loss of configuration. @ashit-rath @riodeuno @somangshu
5. Implement the ongoing defaultValue DX changes for the select and multi-select V2. ([Bug][DX] Use both string and Object values for the default values in select and Multiselect widget #11074) @ashit-rath

[ashit-rath]

Actionables:

• Discussion to remove original_index from the Table selectedRow property @sbalaji1192
• Modify the implementation of the fixed footer @ashit-rath
• Design for the form mode @vasanth-appsmith @dilippitchika
• Implementation of the form mode @ashit-rath
• Discuss support for non-ASCII characters @ashit-rath @sbalaji1192
• Collapse the style sections in property pane @ashit-rath
• QA to continue testing @vivek
• Think about ways to educate users about arrays and objects @dilippitchika

[ashit-rath]

Actionables:

• - Implement support for non-ASCII characters @ashit-rath
• Assess the naming of the property Accessor @dilippitchika
• Documentation efforts @dilippitchika
• Performance evaluation change @ApekshaBhosale

[dilippitchika]

Accessor name change in JSON form

Suggestion to call this an “Alias” as - "a way to refer this field". We are also following the same pattern in the table widget.

CC - @ashit-rath @somangshu

[ashit-rath]

Actionables:

• Discuss edge cases and new features with QA. @ashit-rath @vivekverma2312
• review non-ASCII @sbalaji1192
• documentation @dilippitchika
• Work on QA feedback @ashit-rath
• Awaiting product feedback on the accessor naming convention @ashit-rath @dilippitchika

[somangshu]

• release
  • Final QA feedback
  • test non-ASCII support
  • address any issue from QA feedback