Friday, December 29, 2017

DBMS for Analytics: Risky Business Without Foundation Knowledge, Part 2 (A2)


In this two-part series I alert analysts that correct interpretation and assessment of media/industry claims without being misled requires a good grasp of data fundamentals. In Part 1, I discussed the logical-physical confusion and the erroneous missclassification of DBMSs as relational and non-relational underlying the argument that the latter are superior to the former for analytics applications. In Part 2, I discuss a third misconception behind the claim.

Thursday, December 21, 2017

Three Re-writes with Season's Greetings

The following posts have been re-written to bring in line with the McGoveran formalization and interpretation of Codd's true RDM. Re-reading is strongly recommended.
"But the core Information Principle (IP) of the RDM mandates that all information in a relational database be represented explicitly and in exactly one way -- as values of relation attributes defined on domains. The difference between relation names is, thus, meaningful information, the representation of which violates the IP and the RDM, for which reason it is inaccessible to the DBMS: consider the candidate tuple {v1,v2} -- it is impossible for the DBMS to know to which relation it belongs based on the relation and attribute names because it does not understand semantics!" Database Design Relation Predicates and “Identical Relations”
"Some set defining properties are formed as the disjunction of two or more properties (a kind of relationship between two common properties). These disjuncts, taken together, are meaning criteria. Each meaning criterion (an individual disjunct) induces a partitioning of a set into two subsets, those that meet the criterion and those that do not. Alternatively, we can say that each meaning criterion serves to differentiate a possible subset of a set from other subsets of the set (some of the possible subsets will be disjoint, while others not). Each of the possible subsets of the set is then defined by (“inherits”): The defining properties of the set conjoined with at least one meaning criterion (that or those becoming the defining property, or properties, respectively, specific to the proper subset)." Meaning Criteria and Entity Supertype-Subtypes
"Although they are no longer used, inquiries about them persist and with the current proliferation of non-relational products (e.g., NoSQL, graph DBMSs) there is value in understanding them. The closest the industry came to implementing the RDM is SQL which, despite its poor relational fidelity, proved much superior relative to the complexity and inflexibility of preceding DBMSs. But the rules still expose poor relational fidelity of SQL DBMS's that have not been addressed for four decades, while new RDM violations were introduced.

We offer here our clarifications on the rules. For each rule, we:
  • Explain its intended objective;
  • Offer clarifications, some of which reflect our current understanding of the RDM -- distinct from conventional wisdom -- based on its dual theoretical foundation and a careful analysis of Codd's work;" --Interpreting Codd's 12 Rules



 




Sunday, December 17, 2017

This Week


1. Database truth of the week

"Within the database field, it is common to refer to three “level” of description: conceptual, logical, and physical. Both the logical level and the physical level are formal systems. By contrast, the conceptual level is typically an informal system and refers to the subject of the database.

The conceptual language is a subject language, in the terminology of formal systems. The conceptual level identifies the concepts to be formally represented by the logical and physical levels, and how users think and talk about those concepts. This level corresponds only informally to the so-called “conceptual schema” of earlier approaches to information management, which emphasized the capture of conceptual information using various techniques including diagrams and documentation having various degrees of formality, but not forming a strictly formal system themselves."
-- David McGoveran

2. What's wrong with this database picture?


I re-wrote two older debunkings to bring them in line with the McGoveran formalization and interpretation of Codd's true RDM. Re-reads are recommended.
"Can you have 2 tables, VIEWS and DOWNLOADS, with identical structure in a good DB schema (item_id, user_id, time). Some of the records will be identical but their meaning will be different depending on which table they are in. The "views" table is updated any time a user views an item for the first time. The "downloads" table is updated any time a user downloads an item for the first time. Both of the tables can exist without the other ..."
"I have a database for a school ... [with] are numerous tables obviously but consider these:
CONTACT - all contacts (students, faculty) has fields such as LAST, FIRST, MI, ADDR, CITY, STATE, ZIP, EMAIL;
FACULTY - hire info, login/password for electronic timesheet login, foreign key to CONTACT;
STUDENT - medical comments, current grade, foreign key to CONTACT.
Do you think it is a good idea to have a single table hold such info? Or, would you have had the tables FACULTY and STUDENT store LAST, FIRST, ADDR and other fields? ..."

3. To Laugh or Cry?

"Database design is the structure a database uses to plan, store and manage data. Data consistency is achieved when a database is designed to store only useful and required data ... The outline of the table allows data to be consistent. Cascading also ensures data uniformity ... Optimized relationships ensure efficient database performance ... Overall performance of a database is dependent on the design ... Database normalization, or data normalization, is a technique to organize the contents of the tables for transactional databases and data warehouses ... without normalization, database systems can be inaccurate, slow, and inefficient, and they might not produce the data you expect ... When you normalize a database, you have four goals: arranging data into logical groupings such that each group describes a small part of the whole; ... organizing the data such that, when you modify it, you make the change in only one place; and building a database in which you can access and manipulate the data quickly and efficiently without compromising the integrity of the data in storage ... Sometimes database designers refer to these goals in terms such as data integrity, referential integrity, or keyed data access." --Halil Lacevic, What are the uses or importance of database design? Quora.com

Sunday, December 10, 2017

Conventional Wisdom and True Relational Features

Here's what's wrong with last week's picture, namely:
"Per Date’s AN INTRODUCTION TO DATABASE SYSTEMS, Date & Darwen’s DATABASES, TYPES, AND THE RELATIONAL MODEL, and related references, the features of a relational database are values, types, attributes, tuples, relations, relation-valued variables, operators, and constraints.
  • A type is a set of values and related operators.
  • An attribute is a name, value, type triple.
  • A tuple is a set of attributes.
  • A relation is a set of tuples with a given heading.
  • A relation-valued variable (known as a relvar) is a persistent variable whose time-varying value is a relation." --Dave Voorhis, Computer scientist; lead developer of Rel, a true relational database system, Quora.com

This is more or less the conventional wisdom, which is nothing like the true RDM envisioned by Codd [1].


Sunday, December 3, 2017

DBMS for Analytics: Risky Business Without Foundation Knowledge (Part 1)

A new study finding that "non-relational database management systems now comprising 70% of analytics data sources" attributes their popularity to "superiority" to RDBMSs in satisfying analytics needs. There are good reasons to be skeptical of such findings, but even if this one were true, the arguments advanced in support of the attribution are rooted in the misconceptions due to poor foundation knowledge debunked by this blog (and in more depth at dbdebunk.com). Let's see.

This Week


1. Database Truth of the Week

"A formal system is a systematic way of representing something. We call that something the subject and often refer to it as the “subject system”, although it might not be “systematic” in any sense at all. By contrast we call the formal system the object system -- a system of abstract “objects”.

Representing a subject with a formal system allows us to reason about it without getting trapped in ambiguities, or circular arguments. The formal system becomes a theory about that portion of the subject that has been represented and when that portion is faithfully represented by the theory, we say that portion is a model of the theory.
There are at least three distinct uses of languages necessary to use or apply any formal system that correspond to the three distinct ways in which we need to discuss and use formal systems. It is standard to refer to these three uses by different names and as if they were distinct languages: subject, object and meta-language." --David McGoveran

Ed. Note: In the database context, the subject language expresses the conceptual model; the object language expresses the logical model; the data model is the meta-language that expresses the relationship between object and subject.




2. What's Wrong With This Database Picture?

"Per Date’s AN INTRODUCTION TO DATABASE SYSTEMS, Date & Darwen’s DATABASES, TYPES AND THE RELATIONAL MODEL and related references, the features of a relational database are values, types, attributes, tuples, relations, relation-valued variables, operators, and constraints.
  • A type is a set of values and related operators.
  • An attribute is a name, value, type triple.
  • A tuple is a set of attributes.
  • A relation is a set of tuples with a given heading.
  • A relation-valued variable (known as a relvar) is a persistent variable whose time-varying value is a relation." --Dave Voorhis (Computer scientist; lead developer of Rel, a true relational database system), --What are the features of a relational database?, Quora.com

Sunday, November 26, 2017

What Relations Really Are and Why They Are Important

Note: Some of the References have been re-written to bring them into line with the McGoveran formalization and interpretation [1] of Codd's real RDM -- re-reading is recommended.

Here's what's wrong with the picture of two weeks ago, namely:

"In SQL RDBMSes (such as MS SQL Server and Oracle] tables are permently stored relations, where the column names defined in the data dictionary form the "heading" and the rows are the "tuples" of the relation."

"A relation can be represented by a table in database. A relation in the context of modeling a problem will include the fields and possibly the identification of fields which have relationships with other relations..."

"Put simply, a "relation" is a table, the heading being the definition of the structure and the rows being the data."

"In simple English: relation is data in tabular format with fixed number of columns and data type of each column. This can be a table, a view, a result of a subquery or a function etc."

"A relation is a table, which is a set of data. A table is the result of a query."

--What is a relation in database terminology?, StackOverflow.com

Sunday, November 19, 2017

This Week

1. Database Truth of the Week

"Codd's original 'normal form' in 1969-70 is not equivalent to the current 1NF. The former and 'normalization' were tied to Codd's first definition of join in 1969. Multiple normal forms (1NF-5NF) and 'further normalization' are a consequence of Codd's re-definition of join in 1970 as the one of today. Once 1NF and further normalization to at least 2NF have been introduced, 'the one normal form' makes no longer sense. Thus, there is no way to answer "what is the difference between the normal form and 1NF" without taking into account the definition of join, and -- if defined as we now do -- no way to understand the former, except to say that it was to the 1969-70 join what the 5NF is to the current join.That is one reason I personally strongly believe that while relations are at least in 1NF -- a relation that is not, cannot be represented formally as a relation, nor do the formal operators of the relational algebra work correctly otherwise -- even this is insufficient and formal relations must be in 5NF. Otherwise put, there is no such thing as a non-5NF database relation." --David McGoveran

2. What's Wrong With This Database Picture?

"A relation is a data structure which consists of a heading and an unordered set of tuples which share the same type", according to Wikipedia on 'Relation (database)'."
"In SQL RDBMSes (such as MS SQL Server and Oracle] tables are permanently stored relations, where the column names defined in the data dictionary form the "heading" and the rows are the "tuples" of the relation."
"A relation can be represented by a table in database. A relation in the context of modeling a problem will include the fields and possibly the identification of fields which have relationships with other relations..."
"Put simply, a "relation" is a table, the heading being the definition of the structure and the rows being the data."
"In simple English: relation is data in tabular format with fixed number of columns and data type of each column. This can be a table, a view, a result of a subquery or a function etc."
"A relation is a table, which is a set of data. A table is the result of a query." --What is a relation in database terminology?, StackOverflow.com

Wednesday, November 8, 2017

Data Model: The RDM Is, the E/RM Isn't

Minor touches, 11/17/17.


Note: This is a 11/08/17 rewrite of two older posts to bring them in line with the McGoveran's formalization and interpretation [1] of Codd's true RDM.

Here's what is wrong with the picture of two weeks ago, namely:

"ERM is a data model -- So says Date, Chen, etc. So says the majority of current industry experts. Refer to Date 6th edition p347. With very strong references to Codd (who he worked with), Date elegantly explains the differences between RM and ERM -- but clearly believes both are data models (even allowing for the charitable comment). If we take a RDB as the ultimate target implementation of data, and an ERM (or extended) can correctly design all the artifacts that are implemented, this means it is modelling [sic] the data. Granted, an ERM does not explicitly model some of the non-structural aspects of the original Codd definition.

Out of interest, is there a common Relational Modelling tool, that is not also an ERM tool and models the full Codd definition? There are also several other methods of modeling data -- ERM is more a mechanism to represent the data. If ERMs are used by IT professionals across the world to direct the design and build of the majority of applications guided by standard methodologies, is the view of this argument that these were all build wrongly? Regardless of success? Is the inferred conclusion that only the RM models data, and ERM, [or] any other techniques do not? [If so] that is a little limiting."
  
Chen's E/RM [2] (1976) preceded Codd's definition of a formal data model [3] (1980) and it would be unfair to hold it to that definition. But even if Chen used "unified view of data" in his paper's title, there is no excuse for "the majority of industry experts" claiming, post-1980, that the E/RM is a data model.

Sunday, November 5, 2017

This Week

1. Database Truth of the Week

"Logical (aka. syntactic) validity: Every logical inference (application of any number of rules of inference) from syntactically valid and well-formed premises yields well-formed and syntactically valid consequents (i.e., the rules of inference preserve syntactic validity and well-formedness.) This property is independent of any interpretations of the symbols. A query result is said to be logically valid iff it is derived by any sequence of RA operations on one or more relations.

"Semantic correctness: Every interpretation of the symbols (meaning and truth value assignment) that makes the axioms true, makes the theorems true. When we extend a logical model with semantics (specific to the subject matter and its "business" rules) via constraints, those constraints become axioms that must be true. A query result is said to be semantically correct iff for every assignment of meaning to relations that are the RA operands under which their tuples represent true facts, the tuples of the result relations also represent true facts.
If relations are not in 5NF, query results aren’t guaranteed to be semantically correct. Any update anomaly can appear in a result -- a join might deliver extra tuples that are anomalous. A database design that permits update anomalies does not preserve semantic correctness!" --David McGoveran


2. What's Wrong With This Database Picture?

"If we take a RDB as the ultimate target implementation of data, and an ERM (or extended) can correctly design all the artifacts that are implemented, this means it is modelling the data. Granted, an ERM does not explicitly model some of the non-structural aspects of the original Codd definition.

"ERM is a data model -- So says Date, Chen, etc. So says the majority of current industry experts. Refer to Date 6th edition p347. With very strong references to Codd (who he worked with), Date elegantly explains the differences between RM and ERM – but clearly believes both are data models (even allowing for the charitable comment).

Out of interest, is there a common Relational Modelling tool, that is not also an ERM tool and models the full Codd definition? There are also several other methods of modeling data -- ERM is more a mechanism to represent the data. If ERMs are used by IT professionals across the world to direct the design and build of the majority of applications guided by standard methodologies, is the view of this argument that these were all build wrongly? Regardless of success? Is the inferred conclusion that only the RM models data, and ERM, [or] any other techniques do not? [If so] that is a little limiting." --LinkedIn.com


Monday, October 30, 2017

The Importance of Understanding Classes, Sets, and Relations for Analytics

One of the clearest indications of poor foundation knowledge in data management practice is misuse and abuse of terminology. Many data professionals are inducted into the industry without a formal education, via programming and software tools, and use terms indiscriminately, as jargon, without understanding them. This has produced weak DBMS implementations and poorly designed databases that put the correctness of databased analytics at risk (my forthcoming DBDEBUNK Dictionary of Data Fundamentals is an effort to address this problem.)

Sunday, October 29, 2017

Database Design: What It Is and Isn't

Revised 10/31/17.

Note: Posts starting with this one will be consistent with the TERMINOLOGY page and the DICTIONARY OF DATA FUNDAMENTALS. Fundamental terms -- the grasp of which is necessary for data management practice -- will be boldened. When you encounter one you don't understand, better find out what it means, chances are it's being misused or abused. Once the page is finalized, labels and, time permitting, old posts may also be revised accordingly. 

Reference [9] is an important rewrite and is recommended pre-requisite
for this post that you should read first.

Here's what's wrong with the picture of three weeks ago, namely:
"The term database design can be used to describe many different parts of the design of an overall database system. Principally, and most correctly, it can be thought of as the logical design of the base data structures used to store the data. In the relational model these are the tables and views. In an object database the entities and relationships map directly to object classes and named relationships. However, the term database design could also be used to apply to the overall process of designing, not just the base data structures, but also the forms and queries used as part of the overall database application within the database management system(DBMS).

The process of doing database design generally consists of a number of steps which will be carried out by the database designer. Usually, the designer must:

  • Determine the data to be stored in the database.
  • Determine the relationships between the different data elements.
  • Superimpose a logical structure upon the data on the basis of these relationships.
Within the relational model the final step above can generally be broken down into two further steps, that of determining the grouping of information within the system, generally determining what are the basic objects about which information is being stored, and then determining the relationships between these groups of information, or objects." --Halil Lacevic, What is a Relational Database?, Quora.com
Many problems in database practice are due to failure to grasp what a data model is and the important distinctions between (1) data management and other DBMS functions on the one hand and (2) DBMS and application functions on the other.

The three design steps above are vague, somewhat confused and obscure more than enlighten. They do not reflect the fact that database design is formalization of a conceptual model of reality as relations constrained to be consistent with the business rules the model consists of. 

Sunday, October 22, 2017

This Week

1. Database Truth of the Week

"The original normal form and the later First Normal Form (1) are distinct. In the early 1969 RDM there was only "the normal form" of relations [a term Codd borrowed from FOPL]. It was based on the initial version of the join operation, which was different than today's join. Had 1NF and further normalization to at least 2NF had been introduced then, the normal form would have made no sense, as there would have been then multiple normal forms, which make sense only with the post-1970 join definition currently in use. Thus, there is no way to answer "what is the difference between the original normal form and 1NF?" without taking into account the definition of join, and -- if defined as we now do -- no way to understand the original normal form, except to say that in the context of the original join definition it would correspond to today's Fifth Normal Form (5NF). This is why a relation is really in 5NF by definition, not in 1NF as per current understanding." --David McGoveran



2. What's Wrong With This Database Picture?

"The term database design can be used to describe many different parts of the design of an overall database system. Principally, and most correctly, it can be thought of as the logical design of the base data structures used to store the data. In the relational model these are the tables and views. In an object database the entities and relationships map directly to object classes and named relationships. However, the term database design could also be used to apply to the overall process of designing, not just the base data structures, but also the forms and queries used as part of the overall database application within the database management system(DBMS).

The process of doing database design generally consists of a number of steps which will be carried out by the database designer. Usually, the designer must:

  • Determine the data to be stored in the database.
  • Determine the relationships between the different data elements.
  • Superimpose a logical structure upon the data on the basis of these relationships.
Within the relational model the final step above can generally be broken down into two further steps, that of determining the grouping of information within the system, generally determining what are the basic objects about which information is being stored, and then determining the relationships between these groups of information, or objects." --Halil Lacevic, What is a Relational Database?, Quora.com

Monday, October 9, 2017

This Week


1. Database Truth of the Week

“A DBMS using the RDM for all its functionality would be very limited. The RDM only requires that the declarative data sub-language employed by users for data manipulation -- has power not more expressive than first order predicate logic (FOPL), which implies acceptance of certain limitations on what users can do directly in the language, in return for
Language declarativity and decidability;
Semantic correctness and system-guaranteed logical validity;
Physical and logical independence;
Simplicity.”
                                                  --David McGoveran


2. What's Wrong With This Database Picture?

"The term database design can be used to describe many different parts of the design of an overall database system. Principally, and most correctly, it can be thought of as the logical design of the base data structures used to store the data. In the relational model these are the tables and views. In an object database the entities and relationships map directly to object classes and named relationships. However, the term database design could also be used to apply to the overall process of designing, not just the base data structures, but also the forms and queries used as part of the overall database application within the database management system(DBMS).

The process of doing database design generally consists of a number of steps which will be carried out by the database designer. Usually, the designer must:

  • Determine the data to be stored in the database.
  • Determine the relationships between the different data elements.
  • Superimpose a logical structure upon the data on the basis of these relationships.
Within the relational model the final step above can generally be broken down into two further steps, that of determining the grouping of information within the system, generally determining what are the basic objects about which information is being stored, and then determining the relationships between these groups of information, or objects." 
                             --Halil Lacevic, What is a Relational Database?, Quora.com

Monday, October 2, 2017

Understanding the Division of Labor between Analytics Applications and DBMS

I am coming across, on the one hand, instructions on how to do "analytics with SQL" and, on the other, tools purporting to enable "analytics without SQL." They are an umpteenth iteration of essentially similar ideas during my 30-plus years in data management and reflect common and entrenched fundamental misconceptions that I have documented and analyzed the costly consequences of in my writings and teachings. They will keep repeating, inhibiting genuine progress, as long as data fundamentals are ignored or dismissed. One of the least understood is the distinction between DBMS and application functions.

Sunday, October 1, 2017

Class, Type, Relation and Domain in Database Management

Revised  12/01/17.

This is a 10/01/17 re-rewrite of a 08/12/12 post revised on 12/05/16 to bring it in line with David McGoveran's formal exposition and interpretation[1] of Codd's RDM (as distinct from its common "understanding" in the industry).

Here's what's wrong with last week's picture, namely:

"Our terminology is broken beyond repair. [Let me] point out some problems with Date's use of terminology, specifically in two cases.
  • type = domain: I fully understand why one might equate type and domain, but ... in today's programming practice, type and domain are quite different. The word type is largely tied to system-level (or physical-level) definitions of data, while a domain is thought of as an abstract set of acceptable values.
  • class != relvar: In simple terms, the word class applies to a collection of values allowed by a predicate, regardless of whether such a collection could actually exist. Every set has a corresponding class, although a class may have no corresponding set ... in mathematical logic, a relation is a class (and trivially also a set), which contributes to confusion.
In modern programming parlance class is generally distinguished from type only in that type refers to primitive (system-defined) data definitions while class refers to higher-level (user-defined) data definitions. This distinction is almost arbitrary, and in some contexts, type and class are actually synonymous."
There is, indeed, a huge mess. And, as always, it is rooted in poor foundation knowledge [2], of which the comment itself suffers. 

Friday, September 22, 2017

This Week

1. Database Truth of the Week

“If the data sub-language ... has the power of second order predicate logic (SOPL), expressions are possible that cannot be evaluated (for example, self-referencing expressions) and the formal language is then undecidable, an algorithm to implement a declarative query language is impossible and all hope of physical independence is lost." --David McGoveran


2. What's Wrong With This Database Picture?

"Our terminology is broken beyond repair. [Let me] point out some problems with Date's use of terminology, specifically in two cases.
"type" = "domain": I fully understand why one might equate "type" and "domain", but ... in today's programming practice, "type" and "domain" are quite different. The word "type" is largely tied to system-level (or "physical"-level) definitions of data, while a "domain" is thought of as an abstract set of acceptable values.

"class" != "relvar": In simple terms, the word "class" applies to a collection of values allowed by a predicate, regardless of whether such a collection could actually exist. Every set has a corresponding class, although a class may have no corresponding set ... in mathematical logic, a "relation" is a "class" (and trivially also a "set"), which contributes to confusion.
In modern programming parlance "class" is generally distinguished from "type" only in that "type" refers to "primitive" (system-defined) data definitions while "class" refers to higher-level (user-defined) data definitions. This distinction is almost arbitrary, and in some contexts, "type" and "class" are actually synonymous."

Sunday, September 17, 2017

Database Management: No Progress Without Data Fundamentals

I have recently -- yet again -- been accused in a LinkedIn exchange  of "gibberish without any evidence" and of claiming that "nobody know what they're doing" with databases. I will leave it to readers to judge whether (1) five decades worth of writings and teaching is "no evidence" and (2) my comments in the exchange are gibberish. Here I would like to dare anybody to find claims to that effect in any of my pronouncements. What I did, do and will say is that most data professionals do not know and understand data and relational fundamentals -- an incontrovertible fact proved not just by me[1], but also by others[2,3] and that this inhibits real progress in database management. 

As I wrote two weeks ago:
"The RDM put database management on a formal, scientific foot. Consequently, tool experience and relational terminology are insufficient -- foundation knowledge is necessary. Unfortunately, most data professionals do not possess it, in part because they have been misled by the industry and in part because few go through an education -- as distinct from training -- program that teaches the RDM and teaches it correctly. Consequently, even those with the heart in the right place defend the RDM without a full understanding, their views distorted by what passes for it (stay tuned for a debunking of such a recent example)."
I will now fulfill the promise by debunking just such a "heart-in-the-right-place" defense of the RDM. 

Sunday, September 10, 2017

This Week

1. Database Truth of the Week

“A network is a directed acyclic graph (the "direction" of the transitive relationship) and, thus, amenable to transitive closure (TC). In the Relational Data Model (RDM) that usually means the smallest set that includes all the members that satisfy the transitive relationship in question (for the count of each object type the closure is computed and the count ignores level). While the Relational Data Model (RDM) can handle an important subset of graph theory via special graph domain operators and extensions to the original relational operators, which could be made efficient, it is a very difficult problem. Certain computations on finite sets such as TC are not in general computable in a language based on first order predicate logic (FOPL) that is declarative, decidable and supports physical independence (PI) -- a core relational objective. They require a computationally complete language (CCL) that is imperative and recursive.
A ‘TC function’ can be implemented using a host CCL that returns its result in the form of a relation; then a symbol (i.e., pure syntax) of type relation can be defined in relational algebra that references/invokes that function. From within the algebra it appears to be just a relation and is up to the user to understand what the value of the returned relation means --i.e., that it represents the TC. That understanding/interpretation is outside the algebra and passed to users only via documentation (e.g., some meta-language).” --David McGoveran


2. What's Wrong With This Database Picture?

"I don’t like talking about the relational theory of data. It is absolutely fundamental to any deep understanding of data, but most practitioners get along fine without it. It’s more the implementers of database management systems (DBMSs) who need to understand relational theory, so teaching relational theory to ordinary practitioners is a bit like tormenting people with irrelevant theory before you let them get on with the business at hand. Moreover, some of those who understand relational theory use their knowledge to beat other people over the head with it. I don’t want to be associated with that high-handed approach to this important theory.

But I’ve been goaded. Google made me do it. My attention was drawn to a video put out by some folks at Google, Data Modeling for BigQuery. The video is fine for the most part, but it makes some misstatements about relational theory that just drive me crazy. They repeat commonly accepted misconceptions about relational databases—misconceptions that, unfortunately, have driven some of the “advances” we’ve seen of late in the realm of database technology. There have definitely been some true advances, but some new technology is merely different without being better.
If you’re a practitioner, designing, implementing, and using databases, whether SQL or NoSQL, this won’t matter much to you, although it never hurts to learn a little more about the theory of data. However, if you are a programmer who might be the one who builds the next NoSQL mega-star that will replace decades-old technology, you need to know this, because this knowledge will enable you to blind-side every established DBMS vendor, whether SQL or NoSQL." --Ted Hills, Understand Relational to Understand the Secrets of Data

Friday, September 1, 2017

Don't Confuse/Conflate Database Consistency with Truth

Disregard for foundation knowledge and failure to learn from past mistakes by even data professionals deemed experts inhibit progress in data management and bring back problems already resolved that should be of foremost concern to data analysts. Consider the following:

"Above all else, we count on databases to reflect the truth consistently, or at least to reflect the table data perfectly. The database cannot be blamed when an application (or the end users of an application) place inaccurate data in its tables, but a database must accurately report the data it holds. Therefore, bugs are not all created equal; there are bugs, and there are wrong-rows bugs, bugs that silently misrepresent the data that the tables hold. Even the craziest, most obscure corner case that potentially misrepresents your data should rightly bring a loud chorus: "The emperor has no clothes!" We depend on the database, above all, not to lie."

Sunday, August 27, 2017

Object Orientation, Relational Database Design, Logical Validity and Semantic Correctness

Note: This is a 8/24/17 rewrite of a 5/20/13 post to bring it in line with McGoveran's formal exposition of Codd's RDM [1] and its correct interpretation.

08/25/17: I have added formal definitions of logical validity and semantic correctness. 
09/01/17: Minor revisions. 
09/02/17: Added references. 


Here's what's wrong with last week's picture, namely:
"In my experience, using an object model in both the application layer and in the database layer results in an inefficient system. This are my personal design goals:
- Use a relational data model for storage
- Design the database tables using relational rules including 3rd normal form
- Tables should mirror logical objects, but any object may encompass multiple tables
- Application objects, whether you are using an OO language or a traditional language using structured programming techniques should parallel application needs which most closely correspond to individual SQL statements than to tables or "objects". --LinkedIn.com

Sunday, August 20, 2017

This Week

1. Database Truth of the Week

“... [one] limitation imposed by set semantics is the inability to express the concept of a computer variable to which values can be destructively assigned (or "updated") ... variables can be expressed in logic, but they cannot be expressed in elementary set theory, or first order predicate logic (FOPL) -- the foundations of the RDM. Other, more expressively powerful systems are required. Unfortunately, such powerful formal systems do violence to the RDM and its intent.” --David McGoveran


2. What's Wrong With This Database Picture?

"In my experience, using an object model in both the application layer and in the database layer results in an inefficient system. This are my personal design goals:
- Use a relational data model for storage.
- Design the database tables using relational rules including 3rd normal form
- Tables should mirror logical objects but any object may encompass multiple tables
- Application objects, whether you are using an OO language or a traditionallanguage using structured programming techniques should parallel application needs which most closely correspond to individual SQL statements than to tables or "objects". --LinkedIn.com

Sunday, August 13, 2017

Relational Fidelity, Cursors and ORDER BY

Here's what's wrong with last database picture, namely:
"In a book I am reading (QUERYING SQL SERVER 2012) the author talks about theory of how databases work. He mentions relations, attributes and tuples etc. He frequently stresses the fact that some aspect of T-SQL is not relational. Like in the following excerpt:
"T-SQL also supports an object called a cursor that is defined based on a result of a query, and that allows fetching rows one at a time in a specified order. You might care about returning the result of a query in a specific order for presentation purposes or if the caller needs to consume the result in that manner through some cursor mechanism that fetches the rows one at a time. But remember that such processing isn’t relational. If you need to process the query result in a relational manner--for example, define a table expression like a view based on the query--the result will need to be relational. Also, sorting data can add cost to the query processing. If you don’t care about the order in which the result rows are returned, you can avoid this unnecessary cost by not adding an ORDER BY clause."
I would like to know, since every implementation of SQL pretty much has an ORDER BY clause which makes it non-relational, why does it even matter that (the set after ORDER BY is used) its not relational anymore since its like that everywhere? I can understand if he said it was non-standard, for example using != instead of <> for inequality because that affects portability etc., but I do not understand why something is better being relational. Please enlighten." --stackoverflow.com

Saturday, August 5, 2017

This Week

1. Database Truth of the Week

"Semantic correctness: every interpretation of the symbols (meaning assignment and truth value assignment) that makes the axioms true, makes the theorems true. When we extend a logical data model with semantics (specific to the subject matter and its "business" rules) via constraints, those constraints become axioms that must be true." --David McGoveran

Tuesday, August 1, 2017

Structure, Integrity, Manipulation: How to Compare Data Models


The IT industry operates like the fashion industry: every few years -- and the number keeps getting smaller -- a "new" data technology pops up, with vendors, the trade media and various "experts" all stepping over each other to claim that it'll "revolutionize your business" and unless you jump on the bandwagon, you'll be "left behind." But time and again these prove to be fads lacking a sound foundation. Huge resources are invested in migrations from fad to fad, rather than in productive work (Don't believe the hype about Hadoop usage, Basta, Big Data It's Time to Say Arrivederci). Remember?
"Hadoop seems to take over relational database, as Hbase can store even unstructured data whereas relational data warehouse limits to structured data ... handles traditional structured data just fine, albeit in a different way than a RDBMS ... EDW vendors [will] incorporate Hadoop framework into their core architectures to enable advanced and high performance analytics."

Sunday, July 30, 2017

Integrity Is Not Only Referential: DBMS vs Application Enforced Constraints

Note: This is 07/30/17 rewrite of a 11/11/12 post to bring it line with McGoveran formal exposition of Codd's real RDM [1] and its interpretation.

 

There is nothing wrong per se with the question in last week's picture, namely:
"Can I ask whether people make use of the functionality provided by the database to ensure adequate data quality. Secondly do people apply this retrospectively as seems quite reasonable to me when a data problem is identified and the source data cleaned up--to do so could prevent future errors. There appears to be a tension between this sort of implementation and at least a perception of flexibility as database changes would be required should additional allowable values be required." --LinkedIn.com
except that it's about time such questions are no longer asked. Unfortunately, they are evidence of the persistent lack of foundation knowledge in the industry for more than five decades. Such knowledge would have obviated such questions.

Sunday, July 23, 2017

This Week

1. Database Truth of the Week

"And [AI] weaknesses there are. Watson requires many months of laborious training, as experts must feed vast quantities of well-organized data into the platform for it to be able to draw any useful conclusions. And then it can only draw conclusions based upon the body of data, or ‘corpus’ (plural: ‘corpora’) that it has been trained on. The ‘well-organized’ requirement is especially challenging for Watson, as unprepared data sets are typically insufficient. As a result, Watson customers must hire teams of expert consultants to prepare the data sets, a time-consuming and extraordinarily expensive process." --Is IBM's Watson a Joke?

Sunday, July 16, 2017

Relations and Relationships Part II

--------------------------------------------------------------------------------
I have been using the proceeds from my monthly blog @AllAnalytics to maintain DBDebunk and keep it free. Unfortunately, AllAnalytics has been discontinued. I appeal to my readers, particularly regular ones: If you deem this site worthy of continuing, please support its upkeep. A regular monthly contribution will ensure this unique material unavailable anywhere else will continue to be free. A generous reader has offered to match all contributions, so please take advantage of his generosity. Thanks.
---------------------------------------------------------------------------------
Note: This is a 6/21/17 rewrite of a 4/21/13 post, to bring it in line with McGoveran's interpretation [1] of the true RDM envisioned by Codd . It is the second part of a debunking of a LinkedIn thread (the first part of which was debunked two weeks ago).


Here's what's wrong with last week's picture, namely:
"A conceptual model has no rigorous definition? It is like a sketch of a picture yet to be completed? Or like an outline to a paper to be written or fleshed out? And once the model is rigorously defined, the ad hoc, informal model must be precisely consistent with the underlying model in all its semantics. Are you suggesting that a conceptual model is a precursor to a defined logical (relational) model? Then after the relational model is defined, the conceptual model needs to be a consistent abstraction of the formal logical model. 

What other type(s) of relationships can be explicitly and formally defined in a relational data model? Of course there are many other relationships which can be inferred, such as between an attribute and an entity identifier. Please give me a precise reference to where Codd spoke of relationships [differently than i]n his 1985 piece published in ComputerWorld, [where] he said that the only way to represent a relationship (between entity tables or relations) was through explicitly stored values (i.e., attributes, foreign keys).

What do you mean "Attributes are subsets of domains"? An attribute only exists in the context of a relationship. Something (a domain) is a descriptor of (i.e., is related to) something else (another domain).

What is an "R-table"? What do you mean by a "PICTURE [of a relation]"? There are things and there are views or manifestations/presentations of things. There is the model, and there are various presentations of that model. Is that what you are getting at?"--Gordon Everest, LinkedIn.com

"Do you mean that...relations are defined over types (also known as domains); a type is basically a conceptual pool of values from which actual attributes in actual relations take their actual values. (taken from the SQL AND RELATIONAL THEORY [2009] by Chris Date). I am also not sure about "pointers". Can I define a domain of pointers? There might be an interesting relation over such domain.In addition, what will happen if I define a relation over a set of types, each of which is (another) relation? Lets say that a relation is either defined over types (domains), or defined over a "heading" (or a "definition") of other relations ... and I also try to eliminate identifiers completely". --AT, LinkedIn.com

Sunday, July 9, 2017

This Week

1. Database Truth of the Week

"For the operations of a formal system to have inverses within some specific use of that system (like a specific application):
  • The basic elements must be orthogonal (independent), hence the Principle of Orthogonal Design;
  • The combination of basis elements and operations must be expressive enough to represent every aspect of the subject matter, hence the Principle of Expressively Complete Design;
  • And, at the same time, not so expressive that there is more than one way to express each aspect of the subject matter, hence the Principle of Representation Minimality Design.
The basic elements of a relational database is the relation. Adherence to these principles ensure thatthere is a unique relational expression for every aspect of the subject matter--either a base relation or a derived relation--and if there are two ways to derive a derived relation, then those two expressions are provably equivalent (i.e., the differences are merely syntax and never meaningful)." --David McGoveran

Monday, July 3, 2017

New Paper: Logical Symmetric Access, Data Sub-language, Kinds of Relations, Database Redundancy and Consistency

NEW!!! Paper #2 in the Understanding the Real RDM series. NEW!!!   

The data management field cannot and will not progress without educated and informed data professionals and users. UNDERSTANDING THE REAL RDM is a new series of papers that offers informal access to the forthcoming McGoveran interpretation of the formal real RDM envisioned by E. F. Codd (EFC), contrasts it with the current understanding that emerged after EFC's death and demonstrates the practical implications of the differences.

If you are a thinking data professional interested in understanding the scientific foundations of data management, as opposed to the "cookbook" industry practices, these unique papers are a must read. They give you new insights into the RDM--which I call the real data science--and the practical benefits that fail to materialize due to its misuse and abuse promulgated by DBMS vendors, the trade media, "experts" and poor SQL implementations, all of which ignore its formal theoretical grounding from which all the practical benefits derive. You will also learn to minimize the consequences of deficiencies of and optimize the use of SQL.


Friday, June 30, 2017

Data Meaning: Analytics vs. Data Mining



Years back a client consulted me on a problem: they had an important database the meaning of which they did not know. Could the meaning be reconstructed? The question reflected common poor practices due to poor grasp of data fundamentals.

Data mining is distinct from analytics. The former is aimed at 'finding' meaningful data patterns -- i.e., knowledge 'discovery' -- while the latter derives new knowledge from 'existing' knowledge -- i.e., deduction (see Data, Information, Knowledge Discovery, and Knowledge Representation). 'Sensible' querying of databases to retrieve data for analytic applications and correct interpretation of results without a good grasp of data meaning is a fool's errand. Yet current database practices are extremely deficient in this respect.
Data professionals commonly believe they can infer meaning from sheer inspection of the tabular display of database relations, which is an illusion even for simple ones, particularly when, as is so often the case, they are poorly designed. 

Sunday, June 25, 2017

Relations & Relationships Part I

Note: This is a 10/17 rewrite of a 04/13 post to bring it in line with the McGoveran formalization and interpretation [1] of Codd's true RDM. Some of the references have also been rewritten for the same reason and it is recommended that you re-read them.

Levels of Representation


It is critical to keep levels of representation distinct in one's mind and avoid conceptual-logical conflation (CLC) and logical-physical confusion (LPC) [2], both of which are reflected in the comments. The following terminology [3] helps:

  • Conceptual modeling: Informal language and real world terms (object groups, properties, objects business rules);
  • Logical database design: Formal language and database terms (relations, domains/attributes, tuples, constraints);
  • Physical implementation: Formal language and physical storage and access terms (e.g., files, indexes);

Sunday, June 18, 2017

This Week

1. Database Truth of the Week

"The RDM is a formal system. It has two parts.
  • The Deductive Subsystem: the formal language
  • The Interpretation Subsystem i.e., the application--of that language
Without an interpretation subsystem there is no possibility of applying the formal system and it remains an abstract game of symbols.
Semantics is about applying the RDM to some subject. In effect, what you do is restrict the power of the abstract formalism so that it is more closely aligned with your intended use. That means you are using constraints to limit the vocabulary to the subject matter (and making it finite and usually fairly small) and restricting the possible interpretations that can be used consistently with the resulting subset of the formalism." --David McGoveran

Sunday, June 11, 2017

What Meaning Means: Business Rules, Predicates, Integrity Constraints and Database Consistency


Note: This is a 10/23/17 rewrite of a 7/29/12 post to bring it in line with the McGoveran interpretation [1] of Codd's true RDM.

To understand what's wrong with the picture of two weeks ago, namely:
"If we step back and look at what RDBMS is, we’ll no doubt be able to conclude that, as its name suggests (i.e., Relational Database Management System), it is a system that specializes in managing the data in a relational fashion. Nothing more. Folks, it’s important to keep in mind that it manages the data, not the MEANING of the data! And if you really need a parallel, RDBMS is much more akin to a word processor than to an operating system. A word processor (such as the much maligned MS Word, or a much nicer WordPress, for example) specializes in managing words. It does not specialize in managing the meaning of the words ... So who is then responsible for managing the meaning of the words? It’s the author, who else? Why should we tolerate RDBMS opinions on our data? We’re the masters, RDBMS is the servant, it should shut up and serve. End of discussion." --Alex Bunardzic, Should Database Manage The Meaning?
it helps to consider the quote in the context of another article by the author, "The Myth of Data Integrity", where he reveals that those "DBMS opinions" are constraints (the article has been deleted, but a few comments remain online and are highly recommended for a feel  of the consequences of lack of foundation knowledge).

Thursday, June 1, 2017

Redundancy, Consistency, and Integrity: Derivable Data

Database redundancy can wreak havoc with interpretation of analytics results, but it also poses consistency risks that can affect the correctness of the results themselves. The risks are too underappreciated for effective prevention. Given industry practices, analysts who use databases they did not design, or designed without sufficient foundation knowledge, should be on the alert.


Saturday, May 20, 2017

This Week

1. Database Truth of the Week

"If the physical model does not preserve the properties of the logical model, then it is—by definition—incorrect. That is why the physical model must be derived from the logical model, not the other way around." --David McGoveran


2. What's Wrong With This Database Picture?

"If we step back and look at what RDBMS is, we’ll no doubt be able to conclude that, as its name suggests (i.e. Relational Database Management System), it is a system that specializes in managing the data in a relational fashion. Nothing more. Folks, it’s important to keep in mind that it manages the data, not the MEANING of the data! And if you really need a parallel, RDBMS is much more akin to a word processor than to an operating system. A word processor (such as the much maligned MS Word, or a much nicer WordPress, for example) specializes in managing words. It does not specialize in managing the meaning of the words ... So who is then responsible for managing the meaning of the words? It’s the author, who else? Why should we tolerate RDBMS opinions on our data? We’re the masters, RDBMS is the servant, it should shut up and serve. End of discussion." --Alex Bunardzic, Should Database Manage The Meaning? 

Saturday, May 13, 2017

To Really Understand Integrity, Don't Start with SQL

Here's what's wrong with the picture of two weeks ago, namely:
"Constraints are categorized as follows:
  • Domain integrity Constraints
  • Entity integrity Constraints
  • Referential integrity Constraints
  • Not null
  • Unique
  • Foreign key
  • Check
  • Primary key
Constraints are always attached to a column not a table."
--Dayakar, SQL Constraints
Despite being a critical database function, integrity is insufficiently understood and appreciated. Few practitioners know much beyond just awareness of primary key and referential constraints and question even their necessity.

SQL inhibits understanding, so if you want to really understand integrity, don't start with SQL. Instead, educate yourself on relational integrity and put your SQL DBMS's features in that context--how correctly and completely does it support all the necessary constraints? Then you can (1) make sure that the constraints that it does support are enforced and (2) be aware of those that it does not, the potential risks thereof and the defensive actions to be taken, if necessary. A bonus is that you will finally realize one of the many important differences between SQL DBMSs and a true RDBMS, which are confused in the industry[1]


I dare you to find this information anywhere else!!!!

Sunday, May 7, 2017

This Week Special: The US Sinking Deeper Into the Dark Ages

"...having nothing in them of the feelings or principles of ’76 now look to a single and splendid government of an Aristocracy, founded on banking institutions and monied incorporations under the guise and cloak of their favored branches of manufactures commerce and navigation [add technology], riding and ruling over the plundered ploughman and beggared yeomanry." --Thomas Jefferson

The first Donald Trump who comes along is all it takes to dismantle 200 years of American vaunted democracy and whatever little civilization was there by executive order. The "indestructible" system of checks and balances is collapsing like a house of cards. 


A nation that substitutes job training--if anything--for education, religion for reason and unbounded greed for morality regresses to the dark ages and self destructs.

Wednesday, May 3, 2017

The Necessity of Foreign Keys

Last month I showed how poor grasp of data fundamentals makes it difficult to understand different types of key and the necessity of primary keys (PK). There's another type of key that is poorly understood for the same reason: 
"... [we] wish to make a point. There is something which is bad design/good design/mandatory/optional. Please stop insisting that Primary and Foreign keys are mandatory. They are good design habits but by no means mandatory."
Really? FKs were introduced as a relational solution to the problems of hierarchic database technology of the late 60s. 

Saturday, April 29, 2017

This Week

1. Database Truth of the Week

“… logic—is an analytical theory of the art of reasoning whose goal is to systematize and codify principles of valid reasoning. It has emerged from a study of the use of language in argument and persuasion and it is based on the identification and examination of those parts of language which are essential for these purposes. It is formal in the sense that it lacks reference to meaning. Thereby, it achieves versatility: it may be used to judge the correctness of a chain of reasoning (in particular, a “mathematical proof”) solely on the basis of the form (and not the content) of the sequence of statements, which make up the chain.” --R. R. Stoll

Monday, April 17, 2017

Don't Mix Model with Implementation


Note: This is a rewrite of a 11/20/12 post to bring it in line with the McGoveran interpretation of his formalization [1] of Codd's true RDM.
Revised: 11/15/17
 

Here's what's wrong with the last week's database picture, namely:
"When you design your database tables there are some important things to think of:
- Normalize to remove redundant data
- Use the smallest datatype possible
- Create as few indexes as possible, but not too few
- Avoid redundant indexes
- Every table must have clustered index
...
This is important in a normal database but it is even more important in SQL Azure because you have limited space for every database, your connections may be terminated due to heavy use of resources, you pay for what you use and the data that you transfer. You can use the SQL Azure management portal do design your tables or write the T-SQL statement yourself. The syntax to create a table in SQL azure is the same as in SQL server 2005/2008, but not all options are supported in SQL Azure.
...
CREATE TABLE [dbo].[table1]
 ([id] [int] IDENTITY(1,1) NOT NULL,
  [column1] [NVARCHAR](50) NOT NULL,
  [column2] [NVARCHAR](15) NOT NULL,
  [column3] [TINYNT] NULL,
  CONSTRAINT [pk_table1] PRIMARY KEY CLUSTERED ([id] ASC )
   WITH (-- PAD_INDEX = OFF,
            STATISTICS_NORECOMPUTE = OFF,
            IGNORE_DUP_KEY = OFF,
         -- FILLFACTOR=80,
         -- ALLOW_ROW_LOCKS = ON,
         -- ALLOW_PAGE_LOCKS = ON,
         -- DATA_COMPRESSION=PAGE)"
--HÃ¥kan Winther, A SQL Azure tip a day – Create a table