CSpider 1.0

The Chinese Semantic Parsing and Text-to-SQL Challenge

What is CSpider?

CSpider is a Chinese large-scale complex and cross-domain semantic parsing and text-to-SQL dataset translated from Spider by 2 NLP researchers and 1 computer science student. The goal of the CSpider challenge is to develop natural language interfaces to cross-domain databases for Chinese, which is currently a low-resource language in this task area. It consists of 10,181 questions and 5,693 unique complex SQL queries on 200 databases with multiple tables covering 138 different domains. Following Spider 1.0, in CSpider, different complex SQL queries and databases appear in train and test sets. To do well on it, systems must generalize well to not only new SQL queries but also new database schemas.


CSipder is translated from Spider. However, there can be added challenges. First, structures of relational databases, in particular names and column names of DB tables, are typically represented in English. This adds to the challenges to question-to-DB mapping. Second, the basic semantic unit for denoting columns or cells can be words, but word segmentation can be erroneous. It is also interesting to study the influence of other linguistic characteristics of Chinese, such as zero-pronoun, on its SQL parsing.

CSpider Paper (EMNLP'19)

Getting Started

The data is split into training, development, and unreleased test sets. Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

Details of baseline models and evaluation script can be found on the following GitHub site:

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. Here's a tutorial walking you through official evaluation of your model:

Submission Tutorial

Data Examples

Some examples look like the following:

test image

Have Questions?

Ask us questions at our Github issues page or contact minqingkai@westlake.edu.cn or shiyuefeng@westlake.edu.cn.

We expect the dataset to evolve. We would greatly appreciate it if you could donate us your non-private databases or SQL queries for the project.

Acknowledgement

We thank Tao Yu for sharing the original Spider test set with us, and the anonymous reviewers for their precious comments on this project. Also, we thank Pranav Rajpurkar and Tao Yu for giving us the permission to build this website based on SQuAD and Spider.

Leaderboard - Exact Set Match without Values

Following Spider, we take exact matching evaluation. Instead of simply conducting string comparison between the predicted and gold SQL queries, we decompose each SQL into several clauses, and conduct set comparison in each SQL clause. Please refer to our Github page or the Spider paper and its Github page for more details.

Rank Model Dev Test

1

November 27, 2020
RAT-SQL + GraPPa + Adv

Alibaba

59.7 56.2

2

July 8, 2020
XL-SQL

Anonymous

54.9 47.8

3

November 25, 2020
DG-SQL + Multilingual BERT

University of Edinburgh

https://arxiv.org/abs/2010.11988
50.4 46.9

4

October 10, 2020
RAT-SQL (without schema linking) + Multilingual BERT

Anonymous

41.4 37.3

5

July 15, 2020
RYANSQL + Multilingual BERT

Kakao Enterprise

https://arxiv.org/abs/2004.03125
41.3 34.7

6

July 8, 2020
DG-SQL

Anonymous

35.5 26.8

7

Nov 28, 2019
CN-SQL

oneconnect

22.9 18.8

8

Sep 18, 2019
SyntaxSQLNet (based on Yu et al. (2018a))

Westlake University

https://arxiv.org/abs/1909.13293
16.4 13.3