Though LLM’s capabilities have improved dramatically over the previous couple of years, it’s tough to quantify how good it’s.
So I recall a geometrical downside I discovered on my YouTube channel final yr. This was in June 2024 and I used to be attempting to get the foremost large-scale language mannequin of the time (GPT-4o) to unravel the puzzle. It did not work so it was crucial quite a bit It took loads of effort to discover a resolution and made me surprise how the newest LLM solves the identical puzzle.
puzzle
Right here, I recall briefly how I used to be in search of an answer from LLM on the time. Assume you might have a grid of dots/nodes like this: Within the x and y planes, every node is strictly one unit away from its adjoining nodes. It seems like this
Nicely, that is the query I wished to reply,
What number of distinct squares may be drawn on this diagram?It rapidly grew to become clear that the GPT-4o did not know the reply, so I modified course a bit and requested this as an alternative.
I would love a Python program that plots out all of the squares we are able to
draw on the connected diagram, assuming that the corners of any sq.
should lie on one of many spots on the diagram. Assume every adjoining spot is
1 unit aside in each the x and y instructions. Additionally print out a abstract of
the variety of squares of the identical dimension and what their aspect lengths areA protracted story, I Lastly I obtained the GPT-4o to plot the proper Python-based resolution. Nonetheless, it took me about 2 hours, over 40 repetitions to enhance the solutions, earlier than I discovered one thing that labored.
Observe, have you ever discovered the reply but? I nonetheless cannot imagine that there are 21 squares that may be drawn on this graph.
Interplay with Sonnet 4.5
This brings us again to the current. And I recall the current launch, Claude Sonnet 4.5, the place Anthropic is taken into account the very best coding mannequin presently obtainable. It has been about 16 months since I final checked this, and the way a lot has LLM improved in that point? Let’s have a look at how Sonnet 4.5 offers with this puzzle.
I went to the Anthropic homepage to make use of Sonnet 4.5.
You must see a web page like this.
Click on on the massive black half “Attempt Claude” A button close to the highest proper of the display screen. You may be requested to log in at this level. If you do not have an Anthropic account but, arrange your account now. It is fully free.
Whenever you log in, you can be introduced with a well-known chat-based window the place you’ll be able to work together with Sonnet 4.5 and ask questions on Sonnet 4.5.
Earlier than asking the mannequin to offer a Python program to unravel puzzles, I requested a simple query to see if the mannequin already knew the reply.
I’ve returned to
this grid, I must rely all attainable squares of various
sizes that may be fashioned by connecting the dots.
Let me first establish the grid construction. The dots type:
2 dots within the high row
2 dots within the second row
6 dots within the third row
6 dots within the fourth row
2 dots within the fifth row
2 dots within the sixth row
This seems to be a cross or plus-sign formed grid sample.
Now I am going to rely squares by dimension:
1×1 squares: These are fashioned by any 4 dots that make a unit sq..
Within the horizontal center part (rows 3-4): 5 squares per row × 1 = 5 squares
Within the vertical center part (columns the place the cross intersects): counting
the vertical segments offers us 5 extra squares
Complete: 10 small squares
2×2 squares: These want 4×4 preparations of dots.
Within the horizontal part: 4 squares
Within the vertical part: 4 squares
Overlap in middle: -1 (counted twice)
Complete: 7 squares of dimension 2×2
3×3 squares and bigger: The grid sample is barely 6 dots huge at its most, however the cross form limits the place bigger squares can match. Checking the geometry:
One 3×3 sq. can match within the middle the place the arms of the cross overlap
Complete: 1 sq. of dimension 3×3
Complete: 10 + 7 + 1 = 18 squaresSo that you’re confidently fallacious. I used to be intrigued and intrigued that the mannequin refers to a 3×3 sq. (which isn’t attainable with this grid), so I requested the mannequin to show the sq. on the diagram. That answered:
Wanting on the grid extra rigorously, I must rethink whether or not a 3×3 sq.
really exists.
For a 3×3 sq. to exist, I would wish a 4×4 association of dots
(since a 3×3 sq. requires 4 dots on both sides). Let me test the grid
construction:
The grid seems to be:
Rows 1-2: solely 2 dots every (centered)
Rows 3-4: 6 dots every (full width)
Rows 5-6: solely 2 dots every (centered)
For a 3×3 sq., I would wish 4 consecutive rows and 4 consecutive columns all with dots on the intersection factors. Nonetheless, trying on the cross/plus sample:
The highest solely has 2 columns of dots
The center has 6 columns of dots
The underside solely has 2 columns of dots
Since rows 1-2 and rows 5-6 solely have 2 dots every (not spanning the total width)
, there is not really a 4×4 block of dots wherever on this grid.
I must right my reply: There's NO 3×3 sq. attainable on this grid.
The full must be: 10 (1×1 squares) + 7 (2×2 squares) = 17 squares
I apologize for the error in my preliminary rely. The cross-shaped sample
prevents any sq. bigger than 2×2 from being fashioned.Nicely, I used to be half redeemed with that reply, however principally I did not say accurately that there have been 21 squares in complete.
Last reply
To keep up equity within the 16-month mannequin comparability, I requested Sonnet 4.5 the very same query as I requested on the GPT-4o. I requested this query simply in case.
I would love a Python program that plots out all of the squares we are able to draw on
the connected diagram, assuming that the corners of any sq. should lie on one
of the spots on the diagram. Assume every adjoining spot is 1 unit aside in
each the x and y instructions. Additionally print out a abstract of the variety of
squares of the identical dimension and what their aspect lengths areI considered Sonnet 4.5 for five seconds earlier than producing Python code.
import matplotlib.pyplot as plt
import numpy as np
from itertools import combos
from collections import defaultdict
# Outline the grid factors primarily based on the picture
factors = [
# Top two rows (2 dots each, centered)
(3, 6), (4, 6),
(3, 5), (4, 5),
# Middle two rows (6 dots each, full width)
(1, 4), (2, 4), (3, 4), (4, 4), (5, 4), (6, 4),
(1, 3), (2, 3), (3, 3), (4, 3), (5, 3), (6, 3),
# Bottom two rows (2 dots each, centered)
(3, 2), (4, 2),
(3, 1), (4, 1),
]
def is_square(p1, p2, p3, p4):
"""Test if 4 factors type a sq. and return aspect size if true."""
factors = [p1, p2, p3, p4]
# Calculate all pairwise distances
distances = []
for i in vary(len(factors)):
for j in vary(i + 1, len(factors)):
dist = np.sqrt((factors[i][0] - factors[j][0])**2 +
(factors[i][1] - factors[j][1])**2)
distances.append(dist)
distances.kind()
# For a sq.: 4 equal sides and a pair of equal diagonals
# Diagonals must be sqrt(2) occasions the aspect size
if len(distances) == 6:
sides = distances[:4]
diagonals = distances[4:]
# Test if all sides are equal
if np.allclose(sides, [sides[0]] * 4, atol=1e-6):
# Test if diagonals are equal
if np.allclose(diagonals, [diagonals[0]] * 2, atol=1e-6):
# Test if diagonal = aspect * sqrt(2)
if np.allclose(diagonals[0], sides[0] * np.sqrt(2), atol=1e-6):
return True, sides[0]
return False, 0
def order_square_points(p1, p2, p3, p4):
"""Order factors of a sq. in counterclockwise order ranging from bottom-left."""
factors = [p1, p2, p3, p4]
# Discover centroid
cx = sum(p[0] for p in factors) / 4
cy = sum(p[1] for p in factors) / 4
# Kind by angle from centroid
def angle(p):
return np.arctan2(p[1] - cy, p[0] - cx)
factors.kind(key=angle)
return factors
# Discover all squares
squares = []
squares_by_size = defaultdict(record)
for combo in combos(factors, 4):
is_sq, side_length = is_square(*combo)
if is_sq:
ordered = order_square_points(*combo)
squares.append(ordered)
squares_by_size[round(side_length, 2)].append(ordered)
# Print abstract
print("=" * 50)
print("SUMMARY OF SQUARES FOUND")
print("=" * 50)
total_squares = 0
for side_length in sorted(squares_by_size.keys()):
rely = len(squares_by_size[side_length])
total_squares += rely
print(f"Facet size {side_length}: {rely} sq.(s)")
print(f"nTotal squares: {total_squares}")
print("=" * 50)
# Create visualization
fig, axes = plt.subplots(2, 2, figsize=(14, 14))
axes = axes.flatten()
# Plot 1: All dots
ax = axes[0]
x_coords = [p[0] for p in factors]
y_coords = [p[1] for p in factors]
ax.scatter(x_coords, y_coords, c='black', s=100, zorder=3)
ax.set_xlim(0, 7)
ax.set_ylim(0, 7)
ax.set_aspect('equal')
ax.grid(True, alpha=0.3)
ax.set_title('Unique Grid', fontsize=14, fontweight='daring')
ax.set_xlabel('X')
ax.set_ylabel('Y')
# Plot 2: All squares overlaid
ax = axes[1]
ax.scatter(x_coords, y_coords, c='black', s=100, zorder=3)
colours = plt.cm.rainbow(np.linspace(0, 1, len(squares)))
for sq., colour in zip(squares, colours):
square_closed = sq. + [square[0]]
xs = [p[0] for p in square_closed]
ys = [p[1] for p in square_closed]
ax.plot(xs, ys, c=colour, linewidth=2, alpha=0.6)
ax.set_xlim(0, 7)
ax.set_ylim(0, 7)
ax.set_aspect('equal')
ax.grid(True, alpha=0.3)
ax.set_title(f'All {total_squares} Squares Overlaid', fontsize=14, fontweight='daring')
ax.set_xlabel('X')
ax.set_ylabel('Y')
# Plot 3 & 4: Squares by dimension
plot_idx = 2
for side_length in sorted(squares_by_size.keys()):
if plot_idx >= len(axes):
break
ax = axes[plot_idx]
ax.scatter(x_coords, y_coords, c='lightgray', s=100, zorder=2)
squares_of_size = squares_by_size[side_length]
colours = plt.cm.viridis(np.linspace(0, 1, len(squares_of_size)))
for sq., colour in zip(squares_of_size, colours):
square_closed = sq. + [square[0]]
xs = [p[0] for p in square_closed]
ys = [p[1] for p in square_closed]
ax.plot(xs, ys, c=colour, linewidth=2.5, marker='o', markersize=8)
ax.set_xlim(0, 7)
ax.set_ylim(0, 7)
ax.set_aspect('equal')
ax.grid(True, alpha=0.3)
ax.set_title(f'Squares with aspect size {side_length} ({len(squares_of_size)} complete)',
fontsize=14, fontweight='daring')
ax.set_xlabel('X')
ax.set_ylabel('Y')
plot_idx += 1
plt.tight_layout()
plt.present()At first look, I assumed that Sonnet couldn’t provide you with the proper resolution so rapidly. GPT-4o I had it It took hours. Nonetheless, I put in all of the libraries I wanted for my code, copied it and pasted it right into a Jupyter Pocket book cell, and once I ran it I obtained the next output: The graph has additionally been output, so I’ve omitted it this time.
==================================================
SUMMARY OF SQUARES FOUND
==================================================
Facet size 1.0: 9 sq.(s)
Facet size 1.41: 4 sq.(s)
Facet size 2.24: 2 sq.(s)
Facet size 2.83: 4 sq.(s)
Facet size 3.61: 2 sq.(s)
Complete squares: 21
==================================================
#
# Plus some graphs that I am not exhibiting right here
#That shocked me. The reply was fully correct.
The one factor that this mannequin was not fully right is that it didn’t output plots for every set of squares of various sizes. I simply did 9 1x1s and 4 √2x√2s. I solved this challenge by asking Sonnet to incorporate them as effectively.
Are you able to print the graphs in sq. aspect order. Can also you might have two graphs
aspect by aspect on every "line"That is what it created.
stunning.
abstract
To show how dramatically LLM has made its manner by way of a few yr, I made a decision to revisit the difficult geometric puzzle I attempted to unravel for the primary time utilizing the GPT-4o in June 2024. The puzzle was to put in writing a Python program that discovered and plotted all of the attainable squares on a specific cross-shaped dot grid.
My expertise somewhat over a yr in the past was a battle. It took about 2 hours and over 40 prompts to information the GPT-4o to the proper Python resolution.
As we speak I examined the brand new Claude Sonnet 4.5. Once I first requested the mannequin instantly, I used to be unable to calculate the proper variety of squares. It wasn’t an ideal begin, however in precise checks, Precisely the identical immediate I used it with the GPT-4o.
To my shock, I generated an entire and proper Python resolution. One shot. The generated code not solely discovered all 21 squares, but additionally accurately categorized them by their distinctive aspect size and generated detailed plots to visualise them. One straightforward follow-up immediate was wanted to excellent the plot, however the central challenge was resolved instantly.
Is that the very act of attempting to unravel this puzzle final yr and publishing the outcomes, that implies that it was launched into Internet-o-Sphere, that’s, Anthropic merely crawled it and included it into the mannequin information base? Sure, I believe that is attainable, however why may the mannequin not correctly reply the primary direct query I requested in regards to the complete variety of squares?
For me, this experiment clearly reveals an unbelievable leap in LLM performance. Sixteen months in the past, the foremost fashions on the time had a 2-hour repetition battle, however now the foremost fashions can achieve one shot in 5 seconds.
